Three Things, Tested Three Ways

The confusion comes from treating “the LLM feature” as one indivisible thing. It is three:

The first row is 90% of your code and needs no model at all. Get that boundary right and the rest is manageable.

Put an Interface in Front of the Model

Do not scatter client.Messages.New through your handlers. Define the narrowest interface your code actually needs and depend on that:

// Completer turns a prompt into text. It is deliberately narrow — narrower
// than the SDK — so tests can implement it in five lines.
type Completer interface {
    Complete(ctx context.Context, req Request) (Response, error)
}

type Request struct {
    System   string
    Messages []Message
    Tools    []Tool
}

type Response struct {
    Text       string
    ToolCalls  []ToolCall
    StopReason string
    Usage      Usage
}

This is just the dependency-inversion habit that makes any external service testable, and it earns its keep faster here than almost anywhere else. Two things fall out of it:

Your business logic never imports the SDK. It imports your Request and Response types. When the SDK’s union types change shape, one adapter file changes.

The fake is trivial. No HTTP, no fixtures, no mocking library:

type fakeCompleter struct {
    responses []Response
    err       error
    calls     []Request // recorded for assertions
}

func (f *fakeCompleter) Complete(_ context.Context, req Request) (Response, error) {
    f.calls = append(f.calls, req)
    if f.err != nil {
        return Response{}, f.err
    }
    if len(f.responses) == 0 {
        return Response{}, errors.New("fakeCompleter: no responses left")
    }
    resp := f.responses[0]
    f.responses = f.responses[1:]
    return resp, nil
}

Returning a queue rather than a single value is what lets you test multi-turn loops: the first call returns a tool request, the second returns the final answer.

Test the Loop, Not the Model

An agent loop has plenty of logic worth testing, none of which needs a model. Does it stop when it should? Does it return a result for every tool call? Does it hand a tool failure back rather than aborting?

func TestLoopReturnsResultForFailedTool(t *testing.T) {
    fake := &fakeCompleter{responses: []Response{
        {StopReason: "tool_use", ToolCalls: []ToolCall{
            {ID: "t1", Name: "lookup", Input: []byte(`{"id":"missing"}`)},
        }},
        {StopReason: "end_turn", Text: "I couldn't find that record."},
    }}

    agent := New(fake, map[string]ToolFunc{
        "lookup": func(context.Context, []byte) (string, error) {
            return "", errors.New("not found")
        },
    })

    got, err := agent.Run(context.Background(), "look up record missing")
    if err != nil {
        t.Fatalf("Run() error = %v, want nil", err)
    }
    if got != "I couldn't find that record." {
        t.Errorf("got %q", got)
    }

    // The failure must reach the model as a tool result, not abort the loop.
    second := fake.calls[1]
    result := findToolResult(t, second, "t1")
    if !result.IsError {
        t.Error("tool failure was not marked as an error result")
    }
}

That test catches a real bug — a loop that drops the failed call instead of reporting it — and runs in microseconds with no API key.

The same approach covers the rest of the loop’s contract — the invariants from the agent-loop article: that it stops at MaxIterations, that every tool_use block gets exactly one result, that results come back in one message. Table-driven tests fit this beautifully, since each case is just a different queue of canned responses.

Golden Files for Prompt Assembly

Prompt building is string manipulation, and it drifts. Someone adds a field, reorders a section, changes a heading — and unlike code, a prompt regression produces no compile error and no failing assertion, just slightly worse output that nobody attributes to the change.

Golden files make the diff visible in review:

var update = flag.Bool("update", false, "update golden files")

func TestBuildSystemPrompt(t *testing.T) {
    tests := []struct {
        name   string
        cfg    Config
        golden string
    }{
        {"default", Config{}, "system_default.txt"},
        {"with_tools", Config{Tools: allTools}, "system_with_tools.txt"},
        {"terse_mode", Config{Terse: true}, "system_terse.txt"},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := BuildSystemPrompt(tt.cfg)
            path := filepath.Join("testdata", tt.golden)

            if *update {
                if err := os.WriteFile(path, []byte(got), 0o644); err != nil {
                    t.Fatal(err)
                }
                return
            }

            want, err := os.ReadFile(path)
            if err != nil {
                t.Fatal(err)
            }
            if diff := cmp.Diff(string(want), got); diff != "" {
                t.Errorf("prompt changed (-want +got):\n%s", diff)
            }
        })
    }
}

Run go test ./... -update to accept a change deliberately. The point is not that the golden file is correct — it is that changing it requires saying so out loud, in a diff a reviewer can read.

This also catches the caching bugs from the prompt caching article before they cost you anything. A golden test on the system prompt fails the moment somebody interpolates time.Now() into it, because the output differs on every run.

Which suggests a second, blunter test worth having:

func TestPromptAssemblyIsDeterministic(t *testing.T) {
    // Go randomises map iteration order, so a prompt built by ranging over a
    // map differs run to run — and silently never caches.
    first := BuildSystemPrompt(cfg)
    for i := 0; i < 20; i++ {
        if got := BuildSystemPrompt(cfg); got != first {
            t.Fatalf("prompt is not deterministic on run %d", i)
        }
    }
}

Twenty iterations is enough to make a randomised map order fail essentially every time.

Integration Tests, Behind a Build Tag

You do need a handful of tests that touch the real API — enough to catch an SDK upgrade that changed a union type, a model id that no longer resolves, or streaming that broke. But they cost money and need a key, so they must not run on every go test ./...:

//go:build integration

package llm_test

func TestStreamingReturnsCompleteMessage(t *testing.T) {
    if os.Getenv("ANTHROPIC_API_KEY") == "" {
        t.Skip("no API key; skipping integration test")
    }
    // ... real call, assert on shape rather than content
}

go test ./...                    # fast, free, no key
go test -tags=integration ./...  # the real thing, on demand

Assert on shape, never on wording. resp.Text being non-empty, StopReason being end_turn, Usage.OutputTokens being greater than zero, a streamed message accumulating to the same content as a non-streamed one. Those hold across model versions. “The answer contains the word Paris” does not, and a flaky test that fails once a month teaches your team to ignore failures.

There is a middle option that gets you a long way for free: point the SDK at a local httptest.Server via the ANTHROPIC_BASE_URL environment variable and serve canned JSON. That exercises the real SDK — its parsing, its retry behaviour, its streaming decoder — without a key or a bill. It is the closest analogue to what go-sqlmock does for the database layer: a real driver, a fake server.

It is also the only sane way to test the paths you cannot easily provoke on purpose:

// Rate limiting: does the caller back off, or hammer?
mux.HandleFunc("/v1/messages", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Retry-After", "1")
    w.WriteHeader(http.StatusTooManyRequests)
    fmt.Fprint(w, `{"type":"error","error":{"type":"rate_limit_error","message":"..."}}`)
})

Do the same for a 500, a connection dropped mid-stream, and a malformed body. Those are the failures that actually page you, and they are trivial to simulate and nearly impossible to trigger on demand against the real API.

The Refusal Case Nobody Tests

Worth its own paragraph because it is so easy to miss: a safety refusal comes back as HTTP 200 with StopReason set to refusal. Your error handling never fires. Code that goes from if err != nil straight to reading the first content block treats it as an empty answer.

func TestRefusalIsNotTreatedAsEmptyAnswer(t *testing.T) {
    fake := &fakeCompleter{responses: []Response{
        {StopReason: "refusal", Text: ""},
    }}

    _, err := New(fake, nil).Run(context.Background(), "...")
    if !errors.Is(err, ErrDeclined) {
        t.Errorf("got %v, want ErrDeclined", err)
    }
}

One line in the fake, and you have covered a branch most production code does not have at all. The sentinel-error pattern behind ErrDeclined is the one from error handling in Go.

Evals: For Quality, Not Correctness

Everything above tests whether your code is right. None of it tells you whether the answers are any good. That needs a different instrument, and the mistake is trying to force it into go test.

An eval is a fixed set of inputs, run against the real model, scored rather than asserted:

//go:build eval

func TestClassificationAccuracy(t *testing.T) {
    cases := loadEvalCases(t, "testdata/eval/classification.jsonl")

    var correct int
    for _, c := range cases {
        got, err := classify(context.Background(), realClient, c.Input)
        if err != nil {
            t.Fatal(err)
        }
        if got == c.Want {
            correct++
        } else {
            t.Logf("MISS: input=%q got=%q want=%q", c.Input, got, c.Want)
        }
    }

    accuracy := float64(correct) / float64(len(cases))
    t.Logf("accuracy: %.1f%% (%d/%d)", accuracy*100, correct, len(cases))

    // A floor, not an equality check. Below this, something regressed.
    if accuracy < 0.90 {
        t.Errorf("accuracy %.1f%% below the 90%% floor", accuracy*100)
    }
}

Two things make this useful rather than annoying. The threshold is a floor, not a target — you are detecting regression, not demanding perfection. And the misses get logged, because the list of what it got wrong is the actual output; the pass/fail is almost incidental.

Run evals when you change a prompt, a model, or an effort setting — not on every commit. They cost money and take minutes.

What Not to Do

Do not assert on model wording. strings.Contains(resp, "Paris") passes today and fails after a model update that phrases it differently. It is not testing your code.

Do not set temperature to zero and call it deterministic. Sampling parameters are not even accepted on current models, and identical output was never guaranteed regardless.

Do not mock the SDK’s types. Mocking anthropic.Message and its union blocks is a lot of work to test the adapter you wrote to avoid exactly that. Fake your own interface instead.

Do not let integration tests run by default. A test suite that needs an API key is a test suite that new contributors cannot run, and CI cost that grows with every push.

Do not skip testing the error paths because they are “just the SDK’s job”. Rate limits, refusals and mid-stream disconnects are the failures you will actually see in production, and they are the cheapest things in this article to cover.

Checklist

• A narrow interface between your logic and the SDK; business code never imports the SDK.
• A fake with a queue of canned responses for multi-turn loops.
• Unit tests for the loop’s contract: one result per tool call, errors handed back, iteration cap honoured.
• Golden files for prompt assembly, plus a determinism test.
• Integration tests behind //go:build integration, asserting on shape not wording.
• An httptest.Server for 429s, 500s and truncated streams.
• A test for the refusal path.
• Evals behind their own tag, scored against a floor, misses logged.

Conclusion

“You cannot test LLM code” conflates the model with the code around it. The code around it — the prompt builder, the parser, the loop, the tool handlers, the error branches — is ordinary Go, and it becomes easy to test the moment there is an interface between it and the SDK. Push non-determinism out to the edges, cover the edges with evals scored against a floor, and the rest of your suite stays fast, free and green.

One Invariant, Everything Follows From It

Prompt caching is a prefix match. Any change anywhere in the prefix invalidates everything after it.

That is the whole model. The cache key is derived from the exact bytes of your rendered prompt up to each breakpoint. One byte different at position N — a timestamp, a reordered JSON key, an extra tool in the list — and every cached position at or after N is gone.

The render order matters and is fixed:

tools  →  system  →  messages

Tools render first, at position zero. That has a consequence people discover the expensive way: change the tool list and you have invalidated everything, system prompt and entire conversation included. More on that below.

What It Costs

Two numbers govern whether caching pays:

• A cache read costs about 0.1× the base input price.
• A cache write costs 1.25× for the 5-minute TTL, 2× for the 1-hour TTL.

So with the default 5-minute TTL, two requests already break even: 1.25 + 0.1 = 1.35 against 2.0 uncached. By the third request you are well ahead. With the 1-hour TTL you need three requests to break even, because the write costs double.

Which is why the TTL question is not “how long do I want this cached” but “how far apart do requests sharing this prefix start”:

The subtlety in row one: a read refreshes the entry at no extra cost, and the lifetime is measured from the start of the request. A four-minute generation leaves about one minute for the next request to begin before a five-minute entry expires. For a chat endpoint under steady load, the 5-minute TTL is the right answer and the 1-hour TTL just doubles your write bill.

Making It Work in Go

The syntax is a CacheControl on the last block of whatever you want cached. Because tools render before system, a marker on the final system block caches both:

params := anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Tools:     tools, // deterministic order — see below
    System: []anthropic.TextBlockParam{{
        Text:         systemPrompt,
        CacheControl: anthropic.NewCacheControlEphemeralParam(), // 5-minute default
    }},
    Messages: messages,
}

For the 1-hour TTL:

CacheControl: anthropic.CacheControlEphemeralParam{
    TTL: anthropic.CacheControlEphemeralTTLTTL1h,
},

There is also a top-level CacheControl on MessageNewParams that automatically places a breakpoint on the last cacheable block and moves it forward as the conversation grows. For multi-turn chat that is the right default — no marker bookkeeping, and the growing history caches incrementally.

The robust combination for anything agentic: one explicit breakpoint at the end of the static system prefix, so the expensive shared part has a guaranteed read point no matter what happens later in messages, plus top-level automatic caching for the growing tail.

You get four breakpoints per request, so there is no need to be frugal — place them at genuine stability boundaries.

Where Automatic Caching Is the Wrong Tool

Automatic placement puts the breakpoint at the very end of your prompt. When the prompt ends with something unique per request — a retrieved document, the user’s actual question — that is a pure surcharge: every request writes a new cache entry that nothing will ever read.

The signature is unmistakable once you know it: cache_creation_input_tokens is non-zero on every single request, while cache_read_input_tokens never covers the shared prefix.

The fix is an explicit marker at the end of the shared portion:

Messages: []anthropic.MessageParam{
    anthropic.NewUserMessage(
        anthropic.TextBlockParam{
            Text:         sharedContext, // few-shot examples, retrieved docs
            CacheControl: anthropic.NewCacheControlEphemeralParam(),
        },
        anthropic.NewTextBlock(userQuestion), // no marker — differs every time
    ),
},

Same rule, restated: put the breakpoint where the prompt stops being shared, not where the prompt ends.

The Minimum Prefix, Which Is Not Monotonic

A prompt shorter than the model’s minimum will not cache — no error, no warning, cache_creation_input_tokens simply comes back zero. And the minimum does not move in the direction you would guess as models get newer:

A 3,000-token system prompt caches on Opus 5 and Opus 4.8, and silently does not on Opus 4.6 or Haiku 4.5. If you switched models and your cache hit rate fell off a cliff, this is the first thing to check — and it cuts the other way too: moving to Opus 5 halves the Opus 4.8 minimum, so prompts that were previously too short start caching with no code change at all.

Silent Invalidators

This is the part worth committing to memory, because every one of these is code that looks perfectly reasonable in review.

The Go-specific ones deserve emphasis. Map iteration order in Go is deliberately randomised, so this is not a cache bug that appears under load — it appears on every request, and it is invisible because the rendered prompt is semantically identical each time:

// Bad: iteration order is randomised, so the bytes differ every run.
for name, def := range h.tools {
    tools = append(tools, def)
}

// Good: deterministic.
names := slices.Sorted(maps.Keys(h.tools))
for _, name := range names {
    tools = append(tools, h.tools[name])
}

Sorting keys is a one-line fix that most Go LLM code needs and almost none has.

Injecting Dynamic Context Without Breaking Everything

The usual reason a system prompt has a timestamp in it is that the model genuinely needs to know the date, or the user’s plan tier, or the current mode. The instinct is to template it into the system prompt. Don’t — that is the front of the prefix, and it invalidates everything behind it.

Put dynamic context after the cached history instead. On the newest models there is a first-class channel for this: a system-role message appended to messages, rather than an edit to the top-level system field.

// The top-level system prompt stays byte-identical and stays cached.
// The operator instruction goes after the history, invalidating nothing before it.
messages = append(messages, userTurn)
messages = append(messages, anthropic.MessageParam{
    Role: anthropic.MessageParamRoleSystem,
    Content: []anthropic.ContentBlockParamUnion{
        anthropic.NewTextBlock("Terse mode enabled — keep responses under 40 words."),
    },
})

A message at turn five invalidates nothing before turn five. That is the whole trick.

Two constraints: it must follow a user message and be either the last entry or followed by an assistant turn — it cannot be messages[0], so use the top-level system for the initial prompt. And support is model-dependent; unsupported models return a 400 saying the system role is not supported, so catch that and fall back to putting the instruction in a user turn.

Three Rules That Beat Marker Placement

Fix these before you fiddle with breakpoints.

Freeze the system prompt. No dates, no user names, no modes. It is the front of the prefix and everything downstream depends on it not moving.

Never change tools or model mid-conversation. Tools render at position zero, so adding, removing or reordering one invalidates the entire cache. Caches are also model-scoped, so switching models mid-conversation starts from cold. If you need “modes”, do not swap the tool set — pass the mode as message content.

Forked calls must reuse the parent’s exact prefix. Summarisation passes, sub-agents and side computations usually build their own request. If that fork rebuilds system, tools or model with any difference at all, it misses the parent’s cache completely. Copy them verbatim and append the fork-specific content at the end.

That last one is also the argument against a “cheap model for the easy stuff” cascade, at least as a first move. Caches are per model, so routing between two models forfeits cache reuse across them. Measure the capable model at lower effort before you build the cascade — it is often cheaper and simpler, and it keeps one cache namespace.

Verifying It, Forever

Every response carries the accounting (the same Usage struct I said to log from day one in calling an LLM from Go):

InputTokens is the uncached remainder only — not the prompt size. Total prompt = all three added together. If an agent ran for an hour and InputTokens reads 4K, the rest came from cache; check the sum, not the one field.

In a healthy multi-turn loop you should see, on each request:

• CacheReadInputTokens — the whole prior prefix, growing turn over turn.
• CacheCreationInputTokens — roughly the last assistant turn plus the new input. Small.
• InputTokens — just the tail past the last breakpoint.

If CacheCreationInputTokens is instead close to the full conversation size every time, the prefix is being rewritten upstream of your breakpoint. Go find the timestamp.

And then keep checking. The expensive failure here is never the bad first implementation — it is the regression. Caching works the day you write it, then six weeks later somebody adds a dynamic field to the system prompt or a tool list that stopped being sorted, and every request misses. Nothing fails. Nothing pages. You find out from finance.

So make it a standing assertion, not a one-time look:

func TestSystemPromptStaysCacheable(t *testing.T) {
    // Two identical requests: the second must read from cache.
    first := callModel(t, ctx, fixture)
    if first.Usage.CacheCreationInputTokens == 0 {
        t.Fatalf("nothing was cached: prompt may be under the model minimum")
    }

    second := callModel(t, ctx, fixture)
    if second.Usage.CacheReadInputTokens == 0 {
        t.Errorf("cache miss on an identical prompt — a silent invalidator crept in")
    }
}

That test costs a few cents to run and catches a regression that otherwise runs for months. Put it behind a build tag so it only runs when you mean it — the same treatment I gave integration tests in how to test database interactions in Golang. Better still, put a monitor on the ratio of CacheReadInputTokens to total input tokens and alert when it drops.

Checklist

• Frozen system prompt: no dates, IDs, names or conditional sections.
• Deterministic tool list, sorted by name, identical across users.
• One explicit breakpoint at the end of the static prefix; automatic caching for the tail.
• Breakpoint at the end of the shared portion, not the end of the prompt.
• 5-minute TTL under continuous traffic; 1-hour only for 5–60 minute gaps.
• Prompt above the model’s minimum, which changes between models.
• Dynamic context appended after the history, never templated into the system prompt.
• Forks copy the parent’s system, tools and model verbatim.
• A test or monitor on the usage fields, checked on every change to prompt assembly.

Conclusion

Prompt caching is the rare optimisation with no quality tradeoff: identical output, roughly a tenth of the input cost, and lower latency as a bonus. It is also unusually fragile, because it hinges on byte-exact prefixes and fails completely silently. Treat the prompt-building path the way you would treat a cache key anywhere else in your system — deterministic, stable, and covered by a test — and it mostly takes care of itself.

The Loop, In Full

Here is a complete manual loop. It is worth reading once even if you end up using the SDK’s runner, because everything that goes wrong later is easier to diagnose when you know this shape. It assumes you already have a client and know how to read a response — if not, start with calling an LLM from Go.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"

    "github.com/anthropics/anthropic-sdk-go"
)

func main() {
    client := anthropic.NewClient()

    addTool := anthropic.ToolParam{
        Name:        "add",
        Description: anthropic.String("Add two integers"),
        InputSchema: anthropic.ToolInputSchemaParam{
            Properties: map[string]any{
                "a": map[string]any{"type": "integer"},
                "b": map[string]any{"type": "integer"},
            },
        },
    }
    tools := []anthropic.ToolUnionParam{{OfTool: &addTool}}

    messages := []anthropic.MessageParam{
        anthropic.NewUserMessage(anthropic.NewTextBlock("What is 2 + 3?")),
    }

    for {
        resp, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
            Model:     "claude-opus-5",
            MaxTokens: 16000,
            Messages:  messages,
            Tools:     tools,
        })
        if err != nil {
            log.Fatal(err)
        }

        // Append the assistant turn BEFORE handling the tool calls.
        messages = append(messages, resp.ToParam())

        var toolResults []anthropic.ContentBlockParamUnion
        for _, block := range resp.Content {
            switch variant := block.AsAny().(type) {
            case anthropic.TextBlock:
                fmt.Println(variant.Text)
            case anthropic.ToolUseBlock:
                var in struct {
                    A int `json:"a"`
                    B int `json:"b"`
                }
                // block.Input is raw JSON — parse it, never string-match it.
                if err := json.Unmarshal([]byte(variant.JSON.Input.Raw()), &in); err != nil {
                    log.Fatal(err)
                }
                result := fmt.Sprintf("%d", in.A+in.B)
                toolResults = append(toolResults,
                    anthropic.NewToolResultBlock(block.ID, result, false))
            }
        }

        if resp.StopReason != anthropic.StopReasonToolUse {
            break
        }

        // All results from this turn go back in ONE user message.
        messages = append(messages, anthropic.NewUserMessage(toolResults...))
    }
}

Five things in there are load-bearing, and four of them are easy to get subtly wrong.

resp.ToParam() converts the response into a history entry. You must append the assistant’s turn — including its tool-call blocks — before you send the results, or the next request has results referring to a call that does not exist in the conversation.

Parse the tool input; never pattern-match the raw string. variant.JSON.Input.Raw() gives you the JSON to unmarshal. Current models vary their JSON string escaping (Unicode escapes, escaped forward slashes), so anything doing strings.Contains on the serialised input is a bug waiting for a release.

All tool results go back in a single user message. anthropic.NewUserMessage is variadic for exactly this reason. Splitting results across several messages technically works, and it quietly teaches the model to stop issuing parallel calls — which halves your throughput for no visible reason.

StopReason is the exit condition, not “did I see any tool blocks”. Check it after you have appended the results, not before.

Every tool call needs a result. If the model asked for three tools and you return two results, the next request is malformed. Including for the one that failed — which brings us to the most useful trick in this whole article.

Errors Are Results, Not Exceptions

The instinct when a tool fails is to abort the loop. Usually that is wrong. Hand the failure back to the model as a tool result flagged as an error, and it will very often recover on its own — retry with a corrected argument, try a different tool, or tell the user what went wrong:

out, err := h.run(ctx, variant)
if err != nil {
    // isError = true. The model sees the failure and can adapt.
    toolResults = append(toolResults,
        anthropic.NewToolResultBlock(block.ID, err.Error(), true))
    continue
}
toolResults = append(toolResults, anthropic.NewToolResultBlock(block.ID, out, false))

That third parameter is isError. Getting this right turns a class of hard failures into self-correcting ones.

One caveat worth stating plainly: the error text goes into the model’s context, so do not put a raw database error with connection strings and internal hostnames in there. Return the error you would show a careful external user. This is the same discipline as deciding what a sentinel error exposes at your HTTP boundary, which I covered in error handling in Go.

Let the SDK Drive

Once you understand the loop, you mostly do not want to maintain it. The Go SDK’s tool runner handles the iteration, and generates the JSON schema from your struct tags:

import "github.com/anthropics/anthropic-sdk-go/toolrunner"

type GetWeatherInput struct {
    City string `json:"city" jsonschema:"required,description=The city name"`
}

weatherTool, err := toolrunner.NewBetaToolFromJSONSchema(
    "get_weather",
    "Get current weather for a city",
    func(ctx context.Context, in GetWeatherInput) (anthropic.BetaToolResultBlockParamContentUnion, error) {
        return anthropic.BetaToolResultBlockParamContentUnion{
            OfText: &anthropic.BetaTextBlockParam{
                Text: fmt.Sprintf("The weather in %s is sunny, 22°C", in.City),
            },
        }, nil
    },
)
if err != nil {
    return err
}

runner := client.Beta.Messages.NewToolRunner(
    []anthropic.BetaTool{weatherTool},
    anthropic.BetaToolRunnerParams{
        BetaMessageNewParams: anthropic.BetaMessageNewParams{
            Model:     "claude-opus-5",
            MaxTokens: 16000,
            Messages: []anthropic.BetaMessageParam{
                anthropic.NewBetaUserMessage(anthropic.NewBetaTextBlock("What's the weather in Kyiv?")),
            },
        },
        MaxIterations: 5,
    },
)

message, err := runner.RunToCompletion(ctx)

Note the namespace: this lives under client.Beta.Messages, and the types are the Beta* variants — BetaTextBlock, not TextBlock. Mixing the two is the most common compile error here.

MaxIterations is not optional decoration. Without a ceiling, a model that gets into a retry rut can loop until your context deadline, and you pay for every turn. Set it to the smallest number that lets legitimate work finish.

If you need to inspect or gate each step — approvals, audit logging, a check before a destructive tool runs — you do not have to drop back to a manual loop. The runner exposes NextMessage() and an All() iterator so you can step it and look at each message, and its Params field lets you adjust the next request. Reach for the manual loop only when you want control the runner genuinely does not expose.

Running Tools Concurrently — With a Limit

When the model asks for four tools in one turn, running them sequentially wastes the whole point. But the naive concurrent version is the same mistake Go developers make everywhere else:

// Don't. One turn can ask for many tools; this has no ceiling.
for _, call := range calls {
    go run(call)
}

Use a bounded group. The results still have to come back in one message, in a fixed order, so index into a preallocated slice rather than appending from goroutines:

import "golang.org/x/sync/errgroup"

results := make([]anthropic.ContentBlockParamUnion, len(calls))

g, gctx := errgroup.WithContext(ctx)
g.SetLimit(4) // whatever your slowest downstream can absorb

for i, call := range calls {
    g.Go(func() error {
        out, err := h.run(gctx, call)
        if err != nil {
            // Not a group error: hand it to the model instead.
            results[i] = anthropic.NewToolResultBlock(call.ID, err.Error(), true)
            return nil
        }
        results[i] = anthropic.NewToolResultBlock(call.ID, out, false)
        return nil
    })
}
if err := g.Wait(); err != nil {
    return err
}

messages = append(messages, anthropic.NewUserMessage(results...))

Each goroutine writes one distinct slice element, so no mutex is needed — different elements are different memory. Appending to a shared slice from several goroutines is a different story, and so is writing to a shared map; concurrent map writing and reading in Go has the failure mode in detail.

Notice that a tool failure returns nil from g.Go. Returning the error would cancel gctx and kill the sibling tool calls, when what you actually want is to report that one failure to the model and let the others finish. The full set of tradeoffs around SetLimit and error propagation is in worker pools in Go with errgroup.

Designing the Tools Themselves

The loop is the easy part. Tool design is where agents get good or stay bad.

The description is the API documentation, and its reader is the model. A tool called search described as “searches” will be called wrongly and often. Say what it searches, what it returns, and when not to use it. Most “the agent keeps doing the wrong thing” problems are description problems.

Fewer, broader tools beat many narrow ones. Twenty tools that each wrap one endpoint force the model to plan a long chain and give it twenty chances to pick wrong. One query_orders tool with a few well-named parameters usually outperforms get_order, list_orders_by_user, list_orders_by_date and count_orders.

Constrain the schema. Enums, required fields and explicit types are enforced before your handler runs. Every constraint you express in the schema is a class of invalid call you never have to validate by hand.

Make results terse. Tool results occupy context on every subsequent turn of the loop. Returning a 400-row JSON dump costs you tokens on turn two, turn three and turn four. Return the fields the model needs to decide what to do next, and nothing else.

Be deliberate about side effects. The model will call your tools in orders you did not anticipate. Anything that writes, sends, charges or deletes wants an approval gate — step the runner and confirm — or, at minimum, idempotency so a double call is harmless.

Guardrails That Actually Matter in Production

A tool loop has a cost profile unlike a normal handler: every iteration resends the whole conversation. The bill grows quadratically with the number of turns if you are not careful, and three things keep it honest.

Cap the iterations. MaxIterations on the runner, or a counter in your manual loop. Non-negotiable.

Bound the wall clock. A deadline on the context that covers the whole loop, not each request:

ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
defer cancel()

Cache the prefix. Every turn resends the system prompt and the full history. Without prompt caching you pay full price for all of it, every iteration — this is where agent loops get expensive, and it is the one lever with no quality tradeoff at all. It gets its own article.

Then there is the interaction between concurrency and rate limits. A pool of four tool calls, times however many concurrent user requests, times however many turns each — an agent loop is a very effective way to discover your own rate limits. The token-bucket approach from rate limiting Go APIs works just as well pointed at your own outbound calls as at inbound traffic.

Observability, Or You Are Flying Blind

When a loop misbehaves, you need to see what the model actually saw. Log per iteration:

slog.Info("agent turn",
    "iteration", i,
    "stop_reason", resp.StopReason,
    "tools_called", toolNames,
    "input_tokens", resp.Usage.InputTokens,
    "output_tokens", resp.Usage.OutputTokens,
    "cache_read", resp.Usage.CacheReadInputTokens,
)

With a request-scoped logger carrying the conversation id (the pattern from the slog post), you can pull the entire trajectory of one run out of your logs — which is the difference between “the agent is flaky” and “on turn three it called search with an empty query because the description was ambiguous”.

Pitfalls

Conclusion

The loop is twenty lines and you should write it once by hand, then let the runner own it. After that, the work that actually improves an agent is not loop code at all: sharper tool descriptions, tighter schemas, terser results, a hard iteration cap, and enough logging to reconstruct a bad run. The interesting engineering is in the tools, not the loop around them.

Getting a Client

go get github.com/anthropics/anthropic-sdk-go

import (
    "github.com/anthropics/anthropic-sdk-go"
    "github.com/anthropics/anthropic-sdk-go/option"
)

// Reads ANTHROPIC_API_KEY from the environment.
client := anthropic.NewClient()

// Or pass it explicitly, if you load config yourself.
client := anthropic.NewClient(option.WithAPIKey(key))

Build the client once, at startup, and pass it around. It is safe for concurrent use and holds a connection pool — constructing one per request throws away keep-alive and gives you a fresh TLS handshake every time.

The First Call

resp, err := client.Messages.New(ctx, anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Messages: []anthropic.MessageParam{
        anthropic.NewUserMessage(anthropic.NewTextBlock("Summarise this changelog in three bullets.")),
    },
})
if err != nil {
    return fmt.Errorf("summarise changelog: %w", err)
}

Two things about that snippet are worth slowing down on.

The model is a plain string. The SDK ships typed constants (anthropic.ModelClaudeOpus4_8 and friends), but anthropic.Model is an alias for string, so a model that has no constant yet is passed as its id. Either form compiles; check the SDK release notes before assuming a constant exists for the model you want.

MaxTokens is a ceiling, not a target. It is the point at which generation is cut off mid-sentence, and the model is not told about it. Setting it to 500 because you want a short answer does not produce a short answer — it produces a truncated one. Ask for brevity in the prompt and leave the ceiling generous. For non-streaming requests, something around 16000 keeps you clear of both truncation and the SDK’s HTTP timeout.

Content Is a List of Blocks, Not a String

This is where the first hour usually goes. A response is not resp.Text. It is resp.Content, a slice of union values that can hold text, thinking, tool calls and more:

for _, block := range resp.Content {
    switch variant := block.AsAny().(type) {
    case anthropic.TextBlock:
        fmt.Println(variant.Text)
    case anthropic.ThinkingBlock:
        // Reasoning, when you have asked for it to be shown.
        log.Debug("model reasoning", "text", variant.Thinking)
    }
}

block.AsAny() is the accessor that gets you a concrete type to switch on. Reaching for resp.Content[0].Text and hoping works right up until the day a thinking block or a tool call lands in position zero, at which point you silently return an empty string. Write the type switch once, in a helper, and use it everywhere:

// firstText returns the first text block in a response, or "" if there is none.
func firstText(msg *anthropic.Message) string {
    for _, block := range msg.Content {
        if t, ok := block.AsAny().(anthropic.TextBlock); ok {
            return t.Text
        }
    }
    return ""
}

Thinking, and Why You Probably Want It On

Current Claude models can reason before answering. The recommended mode is adaptive — you do not budget tokens for it, the model decides how much thinking a given request deserves:

adaptive := anthropic.ThinkingConfigAdaptiveParam{}

params := anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Thinking:  anthropic.ThinkingConfigParamUnion{OfAdaptive: &adaptive},
    Messages:  messages,
}

There is no ThinkingConfigParamOfAdaptive helper — you construct the union literal and take the address of the variant, as above. That trips people up because almost every other option in the SDK does have a constructor function.

A word of warning if you are carrying settings over from older code: the fixed thinking budget (ThinkingConfigParamOfEnabled(N)) is gone on current models and returns a 400 rather than being ignored. If you want to spend less, the lever is effort, not a token budget — and effort lives inside output_config, not at the top level of the request.

The counter-intuitive part: on the newest models, turning thinking off is not reliably a cost saving. Lower effort with thinking on generally beats thinking off, and disabling it has failure modes of its own. Leave it on and turn effort down.

Streaming

Anything a user waits for should stream. It is not just perceived speed — a long non-streaming request is also the easiest way to hit an HTTP timeout.

stream := client.Messages.NewStreaming(ctx, anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 64000,
    Messages:  messages,
})

for stream.Next() {
    event := stream.Current()
    switch ev := event.AsAny().(type) {
    case anthropic.ContentBlockDeltaEvent:
        switch delta := ev.Delta.AsAny().(type) {
        case anthropic.TextDelta:
            fmt.Print(delta.Text)
        }
    }
}
if err := stream.Err(); err != nil {
    return fmt.Errorf("stream response: %w", err)
}

Two nested type switches is not the prettiest Go you will write, but the shape is stable: outer switch on the event, inner switch on the delta.

Always check stream.Err(). stream.Next() returning false means “no more events” — it does not tell you whether that was a clean finish or a dropped connection. A loop that ignores Err() will happily serve a truncated answer as if it were complete.

If you want the whole message and the incremental deltas, accumulate as you go. There is no GetFinalMessage() on the Go stream:

stream := client.Messages.NewStreaming(ctx, params)

var message anthropic.Message
for stream.Next() {
    message.Accumulate(stream.Current())
    // ... also forward the delta to the user here
}
if err := stream.Err(); err != nil {
    return err
}
// message.Content is now the complete response.

Raising MaxTokens to 64000 in the streaming example is deliberate. Timeouts stop being the binding constraint once you stream, so you can give the model room.

Streaming to a Browser

Server-sent events are the path of least resistance, and Go’s http.ResponseController makes the flushing straightforward:

func (h *Handler) stream(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("X-Accel-Buffering", "no") // stop nginx buffering the stream

    rc := http.NewResponseController(w)

    stream := h.client.Messages.NewStreaming(r.Context(), params)
    for stream.Next() {
        ev, ok := stream.Current().AsAny().(anthropic.ContentBlockDeltaEvent)
        if !ok {
            continue
        }
        delta, ok := ev.Delta.AsAny().(anthropic.TextDelta)
        if !ok {
            continue
        }

        // SSE data frames must not contain raw newlines.
        payload, _ := json.Marshal(delta.Text)
        if _, err := fmt.Fprintf(w, "data: %s\n\n", payload); err != nil {
            return // client hung up
        }
        if err := rc.Flush(); err != nil {
            return
        }
    }

    if err := stream.Err(); err != nil {
        slog.Error("llm stream failed", "err", err)
        fmt.Fprint(w, "event: error\ndata: {}\n\n")
        _ = rc.Flush()
        return
    }
    fmt.Fprint(w, "event: done\ndata: {}\n\n")
    _ = rc.Flush()
}

Three details that cost me an afternoon each:

• X-Accel-Buffering: no. Without it nginx buffers your stream and delivers the whole thing at the end, which looks exactly like streaming being broken. (Nginx has strong opinions about proxied responses generally — I ran into a related set of them in making Nginx cache cookie aware.)
• JSON-encode the delta. A model can and will emit a newline mid-sentence, and a bare newline terminates an SSE frame.
• Pass r.Context(), not context.Background(). When the user closes the tab, the request context cancels, the SDK aborts the HTTP call, and you stop paying for tokens nobody will read.

Deadlines and Cancellation

Context is not decoration here. It is the only thing standing between a slow model call and a goroutine that lives forever:

ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()

resp, err := client.Messages.New(ctx, params)

Set the deadline against how long the work should take, not a habit. A classification call has no business taking 90 seconds; a long agentic turn on a hard problem might legitimately run for several minutes. If you have not internalised how deadlines propagate through a call chain, understanding Golang context covers the machinery.

The corollary at shutdown: an in-flight model call is exactly the kind of long request that a naive SIGTERM handler will sever. Drain it properly — see graceful shutdown in Go web services.

Errors Worth Distinguishing

The SDK returns typed errors. Use errors.As to get at the status code rather than matching on message strings:

resp, err := client.Messages.New(ctx, params)
if err != nil {
    var apiErr *anthropic.Error
    if errors.As(err, &apiErr) {
        switch apiErr.StatusCode {
        case http.StatusTooManyRequests:
            return fmt.Errorf("rate limited: %w", ErrRetryable)
        case http.StatusBadRequest:
            // Your request is malformed. Retrying will not help.
            return fmt.Errorf("bad request to model: %w", err)
        default:
            return fmt.Errorf("model call failed (%d): %w", apiErr.StatusCode, err)
        }
    }
    // Not an API error: context cancellation, DNS, TLS, connection reset.
    return fmt.Errorf("model call failed: %w", err)
}

Wrapping with %w at each layer is what lets the HTTP boundary decide the status code without every layer needing to know about HTTP — the same pattern I laid out in error handling in Go.

The SDK already retries for you. By default it retries a couple of times on 408, 409, 429, 5xx and connection errors, with backoff. Two consequences people miss:

1. Do not add your own retry loop on top without lowering the SDK’s. Three of yours around two of its is nine attempts and a long tail of latency.
2. Wall-clock can reach timeout × (attempts + 1). Your context deadline is the real budget — set it deliberately, because the retry behaviour will happily use all of it.

A Refusal Is Not an Error

This one surprises people. If a safety classifier declines a request, you get HTTP 200 — a perfectly successful response whose StopReason says the model declined:

if resp.StopReason == anthropic.StopReasonRefusal {
    slog.Warn("model declined",
        "category", resp.StopDetails.Category,
        "explanation", resp.StopDetails.Explanation)
    return "", ErrDeclined
}

err is nil. resp.Content may hold nothing useful. Code that goes straight from if err != nil to reading Content[0] treats this as an empty answer and moves on. Check StopReason before you read content — and note the other values you care about: max_tokens means you were truncated, and tool_use means the model is waiting on you (which is a whole article of its own).

Watch the Usage Numbers

Every response carries a token accounting:

slog.Info("model call",
    "input_tokens", resp.Usage.InputTokens,
    "output_tokens", resp.Usage.OutputTokens,
    "cache_read", resp.Usage.CacheReadInputTokens,
    "cache_write", resp.Usage.CacheCreationInputTokens,
)

Log these from day one. They are the only ground truth about what a feature costs, and CacheReadInputTokens in particular is how you find out that your prompt caching silently stopped working three deploys ago — the failure mode there is not an error, just a bigger invoice. That is its own post, because it is the single biggest lever on what an LLM feature costs to run.

A structured logger pays for itself here: these are five numeric fields per call that you will want to aggregate later, which is exactly the case for structured logging with log/slog.

Checklist

• One client, built at startup, shared across handlers.
• Iterate resp.Content with a type switch; never index blindly into it.
• Adaptive thinking on; tune cost with effort, not by disabling it.
• Stream anything a human waits for, and always check stream.Err().
• r.Context() all the way down, with a deliberate deadline.
• errors.As into *anthropic.Error for status-code branching.
• Do not stack your retries on the SDK’s.
• Check StopReason before reading content — a refusal returns 200.
• Log the usage fields from the first commit.

Conclusion

The API surface is small — one endpoint, one loop, a handful of block types. What makes an LLM call different from any other HTTP call in your service is that it is slow, occasionally non-deterministic, priced per token, and able to succeed while declining to do what you asked. Go gives you good tools for exactly those problems, provided you use the context properly and read the response as the structured thing it is rather than the string you wish it were.

The Token Bucket, in One Paragraph

Picture a bucket that holds b tokens and refills at r tokens per second. Every request takes one token. If the bucket is empty, the request is rejected (or waits). That is the whole model, and it has one property that makes it the right default: b is a burst allowance. A client that has been idle can spend its accumulated tokens all at once, then settles into the steady rate. Real traffic is bursty — a page load firing six API calls should not be punished, while a loop firing six hundred should.

import "golang.org/x/time/rate"

// 10 requests per second, bursts of up to 20.
limiter := rate.NewLimiter(10, 20)

if !limiter.Allow() {
    // over budget
}

Three methods, for three different situations:

rate.Limit is a float, so fractional rates work: rate.Every(time.Minute/100) is 100 per minute, and rate.Limit(0.5) is one request every two seconds. rate.Inf disables limiting entirely, which is handy for a per-plan configuration where some tier is unlimited.

A Global Limiter Is Not Enough

The naive version puts one limiter in front of everything:

var global = rate.NewLimiter(100, 200)

func limit(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !global.Allow() {
            http.Error(w, "too many requests", http.StatusTooManyRequests)
            return
        }
        next.ServeHTTP(w, r)
    })
}

This protects your server but not your users: one aggressive client can still eat the entire global budget and everyone else gets 429s. A global limiter is a useful backstop, not a fairness mechanism. What you want is a limiter per client, with the global one behind it.

Per-Client Limiters

Keep a map from client key to limiter, guarded by a mutex, with a janitor that evicts idle entries so the map does not grow forever.

package ratelimit

import (
    "sync"
    "time"

    "golang.org/x/time/rate"
)

type client struct {
    limiter  *rate.Limiter
    lastSeen time.Time
}

// Store hands out one limiter per key and forgets keys that go quiet.
type Store struct {
    mu      sync.Mutex
    clients map[string]*client
    rate    rate.Limit
    burst   int
    ttl     time.Duration
}

func NewStore(r rate.Limit, burst int, ttl time.Duration) *Store {
    s := &Store{
        clients: make(map[string]*client),
        rate:    r,
        burst:   burst,
        ttl:     ttl,
    }
    go s.cleanup()
    return s
}

// Limiter returns the limiter for key, creating it on first use.
func (s *Store) Limiter(key string) *rate.Limiter {
    s.mu.Lock()
    defer s.mu.Unlock()

    c, ok := s.clients[key]
    if !ok {
        c = &client{limiter: rate.NewLimiter(s.rate, s.burst)}
        s.clients[key] = c
    }
    c.lastSeen = time.Now()
    return c.limiter
}

func (s *Store) cleanup() {
    ticker := time.NewTicker(s.ttl)
    defer ticker.Stop()

    for range ticker.C {
        s.mu.Lock()
        for key, c := range s.clients {
            if time.Since(c.lastSeen) > s.ttl {
                delete(s.clients, key)
            }
        }
        s.mu.Unlock()
    }
}

The mutex is not optional. A bare map[string]*rate.Limiter written from concurrent handlers is a textbook data race, and Go’s runtime will happily crash the process with concurrent map writes — the same failure I dug into in concurrent map writing and reading in Go.

Two design notes. sync.Map is not a better fit here: it is optimised for read-mostly workloads with stable keys, and this map is written on every new client. And an unbounded map is a memory-exhaustion vector if the key is attacker-controlled — hence the TTL. For a hard cap, put an LRU in front of it.

The Middleware

func Middleware(store *Store) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            limiter := store.Limiter(clientKey(r))

            // Reserve, rather than Allow, so we can report Retry-After.
            res := limiter.Reserve()
            if !res.OK() {
                // Burst is smaller than the request size; never satisfiable.
                http.Error(w, "rate limit misconfigured", http.StatusInternalServerError)
                return
            }

            if delay := res.Delay(); delay > 0 {
                // We are not going to wait, so give the token back.
                res.Cancel()

                w.Header().Set("Retry-After", strconv.Itoa(int(math.Ceil(delay.Seconds()))))
                w.Header().Set("RateLimit-Limit", strconv.Itoa(limiter.Burst()))
                w.Header().Set("RateLimit-Remaining", "0")
                w.Header().Set("RateLimit-Reset", strconv.Itoa(int(math.Ceil(delay.Seconds()))))

                w.WriteHeader(http.StatusTooManyRequests)
                json.NewEncoder(w).Encode(map[string]string{
                    "error": "rate limit exceeded",
                })
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

res.Cancel() is the line people forget. Reserve takes the token immediately; if you then decide not to wait, cancelling returns it to the bucket. Skip it and every rejected request still consumes budget, so a client that trips the limit stays locked out far longer than intended.

This plugs into any router the same way as the handlers in my Go middleware example:

store := ratelimit.NewStore(rate.Limit(10), 20, 10*time.Minute)

r := chi.NewRouter()
r.Use(ratelimit.Middleware(store))

Choosing the Client Key

This is where rate limiting is usually got wrong, and it is worth more thought than the algorithm.

Authenticated requests: key on the identity. An API key or user ID is stable, meaningful, and cannot be spoofed once you have verified the token. If you are issuing JWTs — as in user authentication in Go Echo with JWT — the subject claim is your key.

Anonymous requests: key on the IP, carefully. r.RemoteAddr behind a proxy is the proxy’s address, so every user shares one bucket. But blindly trusting X-Forwarded-For is worse: it is a client-supplied header, and anyone can put whatever they like in it to get a fresh bucket per request.

func clientKey(r *http.Request) string {
    // Authenticated callers are keyed on identity.
    if userID, ok := auth.UserFrom(r.Context()); ok {
        return "user:" + userID
    }

    // Only trust the proxy header if the request came from our proxy,
    // and take the address the proxy appended — the rightmost hop.
    ip, _, _ := net.SplitHostPort(r.RemoteAddr)
    if isTrustedProxy(ip) {
        if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
            parts := strings.Split(xff, ",")
            ip = strings.TrimSpace(parts[len(parts)-1])
        }
    }
    return "ip:" + ip
}

The rightmost entry is the one your own proxy added; everything to its left came from the client and is unverifiable. If your platform provides a trusted header — Cloudflare’s CF-Connecting-IP, or the standard Forwarded from a proxy you control — prefer it.

One more refinement: not all endpoints are equal. POST /reports/export might cost a hundred times what GET /health does. AllowN and ReserveN let you charge by cost:

cost := 1
if r.Method == http.MethodPost && strings.HasPrefix(r.URL.Path, "/reports") {
    cost = 25
}
res := limiter.ReserveN(time.Now(), cost)

Just keep the burst at least as large as your most expensive operation, or res.OK() returns false forever and that endpoint becomes permanently unreachable.

Limiting Yourself, Too

Rate limiting is not only defensive. When you are the client of somebody else’s API, respecting their limit proactively beats absorbing 429s and retrying. Wait is built for this:

type Client struct {
    http    *http.Client
    limiter *rate.Limiter
}

func NewClient() *Client {
    return &Client{
        http:    &http.Client{Timeout: 10 * time.Second},
        // The upstream allows 5 requests/second; stay under it.
        limiter: rate.NewLimiter(5, 5),
    }
}

func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    // Blocks until a token is available, or ctx is cancelled.
    if err := c.limiter.Wait(ctx); err != nil {
        return nil, fmt.Errorf("rate limiter: %w", err)
    }
    return c.http.Do(req.WithContext(ctx))
}

Wait returns an error if the context is cancelled or if its deadline arrives before a token would — so a caller that has already given up never sits in the queue. That is the context machinery doing exactly what it is for.

This composes neatly with a bounded worker pool: the pool caps how many requests are in flight, the limiter caps how many start per second. They constrain different things and you usually want both, as I covered in worker pools in Go with errgroup.

Where This Approach Stops Working

Be honest about the limits of an in-process limiter.

It is per instance. Three replicas with a limit of 10/s allow 30/s in total, and a client bouncing between them gets a fresh bucket each time. For a real global limit you need shared state — Redis with a Lua script that does the token accounting atomically, or a limiter at the edge.

It is lost on restart. Every deploy resets every bucket. Usually fine; occasionally not.

It costs you a request. The request still reaches your process, gets routed, and allocates before being rejected. Under a genuine flood, that is exactly the work you cannot afford — which is why volumetric protection belongs at the CDN or load balancer, not in your handler.

My rule of thumb: application limits enforce fairness and per-plan quotas; edge limits absorb abuse. They solve different problems and you want both. The nginx layer is a natural place for the coarse one, and it can be surprisingly nuanced — the cookie-aware caching tricks work the same way for keying limit_req zones.

limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;

location /api/ {
    limit_req zone=api burst=200 nodelay;
    proxy_pass http://backend;
}

Testing It

Rate limiting is easy to test badly, because time.Now() is involved. Keep the rates small and explicit rather than sleeping through real seconds:

func TestLimiterRejectsBurst(t *testing.T) {
    store := ratelimit.NewStore(rate.Limit(1), 3, time.Minute)
    h := ratelimit.Middleware(store)(http.HandlerFunc(
        func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) },
    ))

    codes := make([]int, 5)
    for i := range codes {
        req := httptest.NewRequest(http.MethodGet, "/", nil)
        req.RemoteAddr = "203.0.113.7:1234"
        rec := httptest.NewRecorder()
        h.ServeHTTP(rec, req)
        codes[i] = rec.Code
    }

    // Burst of 3 succeeds, the rest are rejected.
    want := []int{200, 200, 200, 429, 429}
    if !slices.Equal(codes, want) {
        t.Errorf("got %v, want %v", codes, want)
    }
}

Then confirm the behaviour under real load before you trust the number. Pointing a load test at the endpoint and watching the ratio of 200s to 429s tells you whether your limit matches the traffic you actually get — the setup in an easy way to load test your web apps is enough for this.

Checklist

• Per-client limiters, not one global bucket, with a global one as backstop.
• Key on identity when authenticated; on a verified IP otherwise.
• Evict idle limiters so the map cannot grow without bound.
• res.Cancel() whenever you reject instead of waiting.
• Send Retry-After and RateLimit-* headers so good clients can behave.
• Burst at least as large as your most expensive weighted operation.
• Wait(ctx) on the client side of other people’s APIs.
• Volumetric protection at the edge, fairness in the application.

Conclusion

golang.org/x/time/rate is one of those packages that does exactly one thing and does it without ceremony. The algorithm is not the hard part — picking a sensible client key, giving tokens back when you reject, and being clear about what an in-process limiter can and cannot promise is where the real work is. Get those right and a single misbehaving script stops being everybody else’s problem. And if the API you are the client of happens to be a model provider, an agent loop is a remarkably efficient way to find its limits — see tool use in Go.

The Problem With the Obvious Version

// Do not ship this.
func fetchAll(urls []string) {
    var wg sync.WaitGroup
    for _, url := range urls {
        wg.Add(1)
        go func() {
            defer wg.Done()
            fetch(url)
        }()
    }
    wg.Wait()
}

Three separate problems:

1. No limit. len(urls) concurrent requests. The remote service rate-limits you, or your file descriptors run out, or both.
2. No errors. fetch returns one and it goes nowhere.
3. No cancellation. If the caller gives up, every goroutine keeps running to completion.

The concurrency itself is not the mistake — the missing back pressure is.

The Classic Channel Pool

The traditional shape is a jobs channel, a fixed number of workers reading from it, and a results channel:

type job struct {
    ID  int
    URL string
}

type result struct {
    JobID int
    Body  []byte
    Err   error
}

func workerPool(ctx context.Context, jobs []job, workers int) []result {
    jobCh := make(chan job)
    resCh := make(chan result, len(jobs))

    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func(workerID int) {
            defer wg.Done()
            for j := range jobCh {
                body, err := fetch(ctx, j.URL)
                select {
                case resCh <- result{JobID: j.ID, Body: body, Err: err}:
                case <-ctx.Done():
                    return
                }
            }
        }(i)
    }

    // Feed the workers, stopping early if the caller cancels.
    go func() {
        defer close(jobCh)
        for _, j := range jobs {
            select {
            case jobCh <- j:
            case <-ctx.Done():
                return
            }
        }
    }()

    wg.Wait()
    close(resCh)

    out := make([]result, 0, len(jobs))
    for r := range resCh {
        out = append(out, r)
    }
    return out
}

This is worth understanding because you will read it in a lot of codebases, and because it shows the mechanics plainly. Note two things that are easy to get wrong:

• close(jobCh) is the workers’ exit signal. for j := range jobCh ends when the channel closes. Forget the close and wg.Wait() blocks forever.
• Every channel send is paired with <-ctx.Done(). Without that, a worker sending to a full resCh that nobody is reading leaks for the lifetime of the process.

It is also about forty lines to do something the standard extended library does in eight.

The errgroup Version

golang.org/x/sync/errgroup is a sync.WaitGroup that also collects the first error and cancels its siblings. SetLimit turns it into a bounded pool:

import "golang.org/x/sync/errgroup"

func fetchAll(ctx context.Context, urls []string) ([][]byte, error) {
    bodies := make([][]byte, len(urls))

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(10) // at most 10 in flight

    for i, url := range urls {
        g.Go(func() error {
            body, err := fetch(ctx, url)
            if err != nil {
                return fmt.Errorf("fetch %s: %w", url, err)
            }
            // Each goroutine owns exactly one slot: no mutex needed.
            bodies[i] = body
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }
    return bodies, nil
}

That is the whole pool. Behaviour worth knowing:

• g.Go blocks once the limit is reached, until a slot frees up. The for loop becomes its own back pressure — no jobs channel needed.
• errgroup.WithContext returns a derived context that is cancelled the moment any goroutine returns a non-nil error. Shadowing ctx with it, as above, is deliberate: every fetch gets the cancellable one.
• g.Wait() returns the first error, and waits for the rest regardless. Later errors are discarded — if you need all of them, collect them yourself (errors.Join is a good fit, see error handling in Go).
• Writing to bodies[i] is safe without a mutex because each goroutine writes one distinct element. Different elements of a slice are different memory; that is not a data race. Appending to a shared slice, or writing to a shared map, absolutely is — see concurrent map writing and reading in Go for what that failure looks like.

A Note on Loop Variables

The example above relies on Go 1.22’s per-iteration loop variables. Before 1.22, i and url were shared across iterations and every goroutine would see the final values — the single most common concurrency bug in Go. On older versions you must copy them:

for i, url := range urls {
    i, url := i, url // required before Go 1.22
    g.Go(func() error { /* ... */ })
}

Since Go 1.22 the copy is unnecessary. Leaving it in is harmless, and I still write it in code that must build on older toolchains. The loop semantics change was one of the more consequential recent additions to the language — I touched on the surrounding rules in mastering Golang for loops.

Streaming Results Instead of Preallocating

Indexing into a preallocated slice only works when you know the number of jobs up front. For a stream, send results down a channel and read them concurrently:

func processStream(ctx context.Context, in <-chan job, workers int) (<-chan result, func() error) {
    out := make(chan result)

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(workers)

    go func() {
        // Closing `out` after every worker has finished lets the consumer
        // range over it and stop naturally.
        defer close(out)

        for j := range in {
            g.Go(func() error {
                body, err := fetch(ctx, j.URL)
                if err != nil {
                    return fmt.Errorf("job %d: %w", j.ID, err)
                }
                select {
                case out <- result{JobID: j.ID, Body: body}:
                    return nil
                case <-ctx.Done():
                    return ctx.Err()
                }
            })
        }
        _ = g.Wait()
    }()

    return out, g.Wait
}

The consumer ranges over out and then calls the returned function to get the error:

results, wait := processStream(ctx, jobs, 8)
for r := range results {
    save(r)
}
if err := wait(); err != nil {
    return fmt.Errorf("process stream: %w", err)
}

g.Wait() is safe to call more than once — subsequent calls return the same error immediately.

Picking the Limit

There is no universal number, but there is a reliable way to think about it.

CPU-bound work — parsing, hashing, image resizing, compression — saturates at roughly the number of cores. More goroutines just add scheduling overhead:

g.SetLimit(runtime.GOMAXPROCS(0))

I/O-bound work — HTTP calls, database queries, object storage — spends most of its time waiting, so the useful limit is much higher. But it is not “as high as possible”: it is whatever the slowest downstream dependency can absorb. If your database pool has 25 connections, a pool of 200 workers means 175 goroutines queueing on a mutex inside database/sql while your latency graph climbs.

// Match the constraint that actually binds.
g.SetLimit(db.Stats().MaxOpenConnections)

For outbound HTTP, remember that Go’s default transport keeps only 2 idle connections per host. Exceed that and you are opening a fresh TCP connection — plus a TLS handshake — for each extra request:

transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConnsPerHost = 50
transport.MaxConnsPerHost = 50
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

Then set the pool limit to match. Tuning one without the other gets you nothing.

Whatever you pick, measure it. Run the job at 5, 10, 25 and 50 and look at total wall time and downstream latency — the fastest setting for your batch is often the one that makes everything else on the system slower. Load testing is the honest way to find out; I wrote about a lightweight setup in an easy way to load test your web apps.

Not Every Goroutine Belongs in a Pool

A pool is for a batch of similar work. Some situations want something else:

Waiting on several different things at once — no limit needed, just a group:

g, ctx := errgroup.WithContext(ctx)

var user *User
var orders []Order

g.Go(func() (err error) { user, err = loadUser(ctx, id); return })
g.Go(func() (err error) { orders, err = loadOrders(ctx, id); return })

if err := g.Wait(); err != nil {
    return nil, fmt.Errorf("load profile: %w", err)
}

Three sequential 100ms calls become one 100ms call. This is the highest-value use of errgroup in a typical request handler, and it needs no pool at all.

Work that must happen in order — a pool is the wrong shape entirely; you want a single consumer, like the simple queue implementation I wrote about earlier.

Fire-and-forget background work — resist it. A goroutine started in a request handler outlives the request, holds whatever it captured, and will be killed mid-flight when the process shuts down. If it matters, it belongs in a durable queue; if it does not, do it inline. The same reasoning applies at shutdown time, which I covered in graceful shutdown in Go web services.

Only trying if there is capacity — TryGo starts the goroutine only if a slot is free, and reports whether it did:

if !g.TryGo(func() error { return prefetch(ctx, url) }) {
    // Pool is busy; skip this optional work rather than blocking.
    metrics.PrefetchSkipped.Inc()
}

Pitfalls

Panic recovery is worth spelling out, because one bad input taking down the whole process is a common way for batch jobs to fail:

g.Go(func() (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("panic processing %s: %v", url, r)
        }
    }()
    return process(ctx, url)
})

The named return value err is what makes this work — the deferred function assigns to it after the panic is recovered.

Conclusion

The pattern is small: pick a limit that matches your real bottleneck, use errgroup.WithContext so failures cancel their siblings, return errors instead of logging them, and give every blocking operation a context. Most of the time that is eight lines and no channel plumbing at all. Save the hand-rolled channel pool for the cases where you genuinely need to stream results or vary the shape of the work — and when you do write one, remember to close the jobs channel.

One place this pattern turns up more than you would expect: running the tool calls an LLM asks for, several at a time but not unboundedly — see tool use in Go.

What Actually Happens on Shutdown

When your orchestrator wants a container gone, it sends SIGTERM and starts a countdown. If the process is still alive when the countdown ends, it gets SIGKILL, which cannot be caught.

A Go program with no signal handling takes the default action for SIGTERM: immediate termination. Every open connection is severed. Any request that was 90% done is simply gone — the client sees a reset, your retry budget takes the hit, and if that request was a payment you now have a support ticket.

Graceful shutdown means using the window between SIGTERM and SIGKILL to:

1. Stop accepting new connections.
2. Let in-flight requests finish.
3. Drain background workers.
4. Close databases, caches and queues.
5. Exit before the countdown runs out.

The Twenty Lines

package main

import (
    "context"
    "errors"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{
        Addr:              ":8080",
        Handler:           newRouter(),
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       15 * time.Second,
        WriteTimeout:      30 * time.Second,
        IdleTimeout:       60 * time.Second,
    }

    // ctx is cancelled the first time we receive SIGINT or SIGTERM.
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    go func() {
        slog.Info("listening", "addr", srv.Addr)
        // ListenAndServe always returns a non-nil error; ErrServerClosed
        // is the expected one after Shutdown.
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            slog.Error("listen failed", "err", err)
            os.Exit(1)
        }
    }()

    <-ctx.Done()
    stop() // restore default handling: a second Ctrl-C now kills us
    slog.Info("shutdown signal received, draining")

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        slog.Error("graceful shutdown failed, forcing close", "err", err)
        _ = srv.Close()
    }

    slog.Info("shutdown complete")
}

That is the whole pattern. A few details are load-bearing:

signal.NotifyContext instead of a channel. Since Go 1.16 this gives you a context.Context that cancels on the listed signals, which composes with everything else that already takes a context. If contexts and cancellation are new to you, understanding Golang context is the background reading.

Calling stop() after the first signal. It restores the default signal behaviour, so an impatient operator pressing Ctrl-C a second time gets an immediate exit instead of being ignored.

A fresh context for Shutdown. Deriving it from ctx would be a bug: ctx is already cancelled, so Shutdown would return instantly and drain nothing.

Checking for ErrServerClosed. ListenAndServe returns it on a clean shutdown. Treating that as a failure produces a scary log line on every normal deploy.

What Shutdown Does and Does Not Do

Server.Shutdown closes all open listeners, closes idle connections, and then waits for active ones to become idle. It returns when everything is drained or when its context expires — whichever comes first.

What it does not cover:

• Hijacked connections, including WebSockets. Shutdown does not wait for them, and it does not close them. You have to track and close them yourself.
• Background goroutines you started outside the request path. Nothing knows about them.
• Long-polling or streaming responses. These are “active” for as long as they stream, so they will hold the drain open until your timeout fires.

For WebSockets, the usual approach is to register a callback that broadcasts a close frame:

srv.RegisterOnShutdown(func() {
    hub.CloseAll(websocket.CloseServiceRestart, "server restarting")
})

RegisterOnShutdown callbacks run in their own goroutines as soon as Shutdown starts, so they get the whole drain window to do their work.

Draining Background Workers Too

Most real services do more than serve HTTP. If you have consumers, cron loops, or a queue like the one in my simple queue implementation, they need to finish too. errgroup keeps this readable:

import "golang.org/x/sync/errgroup"

func run(ctx context.Context) error {
    srv := &http.Server{Addr: ":8080", Handler: newRouter()}
    queue := NewQueue()

    g, gCtx := errgroup.WithContext(ctx)

    // 1. Serve HTTP.
    g.Go(func() error {
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            return fmt.Errorf("http server: %w", err)
        }
        return nil
    })

    // 2. Consume the queue until the group context is cancelled.
    g.Go(func() error {
        return queue.Consume(gCtx)
    })

    // 3. When anything cancels gCtx — a signal, or a failure in another
    //    goroutine — drain the HTTP server.
    g.Go(func() error {
        <-gCtx.Done()

        shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
        defer cancel()

        return srv.Shutdown(shutdownCtx)
    })

    return g.Wait()
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    if err := run(ctx); err != nil {
        slog.Error("service stopped", "err", err)
        os.Exit(1)
    }
    slog.Info("service stopped cleanly")
}

The nice property here is that failure propagates in both directions. A signal drains the HTTP server; a fatal error in the queue consumer also drains the HTTP server, because errgroup.WithContext cancels gCtx as soon as any goroutine returns an error.

The order of shutdown matters, and it is the reverse of startup: stop accepting work, finish what you have, then close the things that work depends on. Closing your database pool before draining HTTP guarantees a burst of errors from requests that were nearly done.

// After g.Wait() returns, nothing is still using these.
defer db.Close()
defer redis.Close()

The Load Balancer Problem

Here is the part that surprises people: even a perfectly graceful process can drop requests.

Between the moment your pod receives SIGTERM and the moment the load balancer stops sending it traffic, there is a gap. Endpoint updates propagate asynchronously — through the API server, to kube-proxy or an ingress controller, and finally to the actual routing table. During that gap the balancer is still sending new connections to a server that has already closed its listener. Those connections are refused.

The fix is to keep serving for a few seconds after the signal arrives:

<-ctx.Done()
slog.Info("signal received, waiting for load balancer to deregister")

// Keep serving while the endpoint removal propagates.
time.Sleep(5 * time.Second)

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)

It feels wrong to sleep on purpose, but it is the standard remedy, and Kubernetes has a hook for exactly this so you do not need it in your code:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 45

preStop runs before SIGTERM is sent, while the pod is already being removed from the endpoints list. By the time your process sees the signal, traffic has stopped arriving.

Whichever way you do it, keep the arithmetic straight:

preStop sleep (5s) + drain timeout (20s) + close time (2s) < terminationGracePeriodSeconds (45s)

If the total exceeds the grace period, you get SIGKILL mid-drain and you are back where you started. Give yourself real headroom — the default grace period is 30 seconds, which is not much once a slow request is in flight.

A readiness probe that starts failing on SIGTERM achieves the same thing more precisely:

var ready atomic.Bool

func init() { ready.Store(true) }

// /readyz
func readyz(w http.ResponseWriter, r *http.Request) {
    if !ready.Load() {
        http.Error(w, "shutting down", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

// On signal, before draining:
ready.Store(false)

Keep /healthz (liveness) returning 200 the whole time — if liveness fails during shutdown, the kubelet may kill the container instead of letting it drain.

Docker Gotchas

Two container-level mistakes will silently defeat everything above.

Shell-form CMD makes your process PID 2. Written as CMD ./server, Docker runs /bin/sh -c ./server. The shell is PID 1, receives SIGTERM, and does not forward it. Your server never hears a thing.

# Wrong — signals stop at the shell
CMD ./server

# Right — exec form, your binary is PID 1
CMD ["./server"]

docker stop waits 10 seconds by default. If your drain window is 20 seconds, you will be killed halfway through. Raise it: docker stop -t 45, or stop_grace_period: 45s in Compose.

Verifying It Works

Do not take it on faith — this is easy to test. Add a slow endpoint, start a request, and signal the process mid-flight:

mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
    select {
    case <-time.After(10 * time.Second):
        w.Write([]byte("finished\n"))
    case <-r.Context().Done():
        // The client gave up; Shutdown does not cancel request contexts.
        return
    }
})

./server &
curl -s localhost:8080/slow &     # starts a 10s request
sleep 1
kill -TERM %1                     # signal while it is in flight

A correct implementation prints finished after ten seconds and then exits. A broken one prints nothing and the curl reports a reset connection.

For the same check under real traffic, point a load test at the service and restart it mid-run — the technique from an easy way to load test your web apps works well here. With graceful shutdown in place your error count during a restart should be exactly zero; without it, you will see the exact number of requests that were in flight.

Note what Shutdown does not do: it does not cancel r.Context() for in-flight requests. That is deliberate — the request should be allowed to complete. It also means a handler with no timeout of its own can hold the drain open until your shutdown context expires, which is why the WriteTimeout in the first example matters.

Checklist

• signal.NotifyContext for SIGINT and SIGTERM.
• srv.Shutdown with its own fresh, bounded context.
• errors.Is(err, http.ErrServerClosed) treated as success.
• Background workers cancelled through the same context, drained before dependencies close.
• Dependencies closed last, in reverse order of startup.
• Readiness probe flipped to failing before the drain begins.
• preStop hook or a deliberate sleep to cover load balancer propagation.
• Grace period comfortably larger than the sum of your timeouts.
• Exec-form CMD in the Dockerfile.
• A test that proves an in-flight request survives a SIGTERM.

Conclusion

Graceful shutdown is one of those features nobody notices when it works — which is precisely the point. Twenty lines in main, a couple of timeouts that add up correctly, and a container that actually forwards signals will turn every deploy from a small burst of errors into a non-event. It is the cheapest reliability win available to a Go service, and worth adding before the next release rather than after the next incident.

Why Structured Logs Beat Formatted Strings

Here is a line the old log package might produce:

2026/08/04 10:15:02 user 42 checkout failed after 1.2s: payment declined

It reads fine. Now try answering “how many checkouts failed for users on the EU cluster last Tuesday between 14:00 and 15:00?” You are writing a regular expression.

The same event as structured data:

{"time":"2026-08-04T10:15:02.113Z","level":"ERROR","msg":"checkout failed",
 "user_id":42,"duration_ms":1204,"reason":"payment declined","cluster":"eu"}

Now it is a query. Every log aggregator — Loki, Elasticsearch, CloudWatch, BigQuery — indexes those fields directly. The point of structured logging is not prettier output; it is that your logs become a queryable dataset.

The Smallest Useful Setup

package main

import (
    "log/slog"
    "os"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: slog.LevelInfo,
    }))

    // Make it the package-level default so slog.Info et al. use it too.
    slog.SetDefault(logger)

    slog.Info("service started", "port", 8080, "env", "production")
}

Output:

{"time":"2026-08-04T10:15:02.09Z","level":"INFO","msg":"service started","port":8080,"env":"production"}

The variadic arguments are alternating keys and values. If you prefer something the compiler can check, use typed attributes instead:

slog.Info("service started",
    slog.Int("port", 8080),
    slog.String("env", "production"),
)

Both forms are fine, and you can mix them. The typed form costs a little more typing and saves you from the classic odd-number-of-arguments bug, where a stray value ends up under the key !BADKEY.

For local development, swap the handler and keep everything else:

var handler slog.Handler
if os.Getenv("ENV") == "development" {
    handler = slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})
} else {
    handler = slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
}
slog.SetDefault(slog.New(handler))

Levels, and Changing Them Without a Redeploy

slog has four built-in levels — Debug (-4), Info (0), Warn (4), Error (8) — as plain integers, so you can define your own in between if you really need to.

The more useful trick is a LevelVar, which lets you change the level at runtime:

var logLevel = new(slog.LevelVar) // defaults to Info

func main() {
    slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: logLevel,
    })))

    // Flip to debug on demand — from a signal handler, an admin endpoint,
    // or a config watcher.
    http.HandleFunc("/debug/level", func(w http.ResponseWriter, r *http.Request) {
        var l slog.Level
        if err := l.UnmarshalText([]byte(r.FormValue("level"))); err != nil {
            http.Error(w, "bad level", http.StatusBadRequest)
            return
        }
        logLevel.Set(l)
        slog.Warn("log level changed", "level", l)
    })
}

Being able to turn on debug logging for two minutes on a misbehaving pod, without a deploy, has saved me more time than any other logging feature. Put that endpoint behind authentication — the router access permission patterns I wrote about earlier work well for exactly this kind of internal route.

Attaching Context With With

logger.With returns a new logger that carries the given attributes on every subsequent call. This is how you stop repeating yourself:

// Once, at construction:
type Worker struct {
    log *slog.Logger
}

func NewWorker(id int, queue string) *Worker {
    return &Worker{
        log: slog.Default().With("worker_id", id, "queue", queue),
    }
}

func (w *Worker) process(job Job) {
    // Every line from this worker carries worker_id and queue automatically.
    w.log.Info("job started", "job_id", job.ID)
    // ...
    w.log.Info("job finished", "job_id", job.ID, "duration_ms", 42)
}

With does the attribute formatting work once, at call time, rather than on every log line — so a long-lived logger built with With is cheaper than passing the same attributes repeatedly.

Use WithGroup when you want to namespace a block of fields:

log := slog.Default().WithGroup("http")
log.Info("request", "method", "GET", "path", "/users")
// {"level":"INFO","msg":"request","http":{"method":"GET","path":"/users"}}

Or group inline, for one call:

slog.Info("upstream call",
    slog.String("service", "billing"),
    slog.Group("response",
        slog.Int("status", 502),
        slog.Duration("latency", 1200*time.Millisecond),
    ),
)

Request-Scoped Logging

The pattern that pays off most in a web service: put a logger carrying the request ID into the request context, then let every layer below pull it out. If you have not used context.Context for this kind of request-scoped data before, understanding Golang context covers the rules — including why the key must be an unexported type.

package logging

import (
    "context"
    "log/slog"
)

type ctxKey struct{}

// Into returns a copy of ctx carrying logger.
func Into(ctx context.Context, logger *slog.Logger) context.Context {
    return context.WithValue(ctx, ctxKey{}, logger)
}

// From returns the logger stored in ctx, or the default logger.
func From(ctx context.Context) *slog.Logger {
    if l, ok := ctx.Value(ctxKey{}).(*slog.Logger); ok {
        return l
    }
    return slog.Default()
}

The middleware that fills it in — a close cousin of the handler-wrapping in my Go middleware example:

func Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        requestID := r.Header.Get("X-Request-Id")
        if requestID == "" {
            requestID = uuid.NewString()
        }

        log := slog.Default().With(
            "request_id", requestID,
            "method", r.Method,
            "path", r.URL.Path,
        )

        start := time.Now()
        rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}

        next.ServeHTTP(rec, r.WithContext(logging.Into(r.Context(), log)))

        log.Info("request completed",
            "status", rec.status,
            "duration_ms", time.Since(start).Milliseconds(),
        )
    })
}

// statusRecorder remembers the status code written by the handler.
type statusRecorder struct {
    http.ResponseWriter
    status int
}

func (r *statusRecorder) WriteHeader(code int) {
    r.status = code
    r.ResponseWriter.WriteHeader(code)
}

Now any function that already takes a context.Context can log with full request context and no extra parameters:

func (s *Store) User(ctx context.Context, id int64) (*User, error) {
    logging.From(ctx).Debug("loading user", "user_id", id)
    // ...
}

Every line from that request — across every layer — shares a request_id. Tracing one user’s bad afternoon becomes a single filter in your log viewer.

Redacting Secrets With ReplaceAttr

HandlerOptions.ReplaceAttr runs on every attribute before it is written. It is the right place to enforce policy centrally instead of trusting each call site.

var sensitive = map[string]bool{
    "password": true, "token": true, "authorization": true,
    "api_key": true, "secret": true, "set-cookie": true,
}

handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level:     slog.LevelInfo,
    AddSource: true, // include file:line — useful for errors
    ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
        if sensitive[strings.ToLower(a.Key)] {
            return slog.String(a.Key, "[REDACTED]")
        }
        // Rename "time" to "timestamp" to match the rest of our pipeline.
        if a.Key == slog.TimeKey && len(groups) == 0 {
            a.Key = "timestamp"
        }
        return a
    },
})

For types you control, implementing LogValuer is even better — the value redacts itself wherever it is logged:

type Password string

// LogValue implements slog.LogValuer so a Password never reaches a log sink.
func (Password) LogValue() slog.Value {
    return slog.StringValue("[REDACTED]")
}

slog.Info("login attempt", "user", "alex", "password", Password("hunter2"))
// {"level":"INFO","msg":"login attempt","user":"alex","password":"[REDACTED]"}

LogValuer is also how you log a struct compactly. Give your User type a LogValue that returns a group with just the ID and role, and you never accidentally dump an entire record — with its email address and hashed password — into a log line.

Logging Errors Properly

Log the error value, not a formatted string, and let the handler decide how to render it:

if err != nil {
    slog.Error("checkout failed", "err", err, "user_id", userID)
    return fmt.Errorf("checkout: %w", err)
}

Two habits worth keeping:

• Log once, at the boundary. If you log here and return the error, every caller up the stack logs it again. That is the same rule I covered in error handling in Go, and slog does not change it.
• Do not log context.Canceled as an error. A user closing a tab is not an incident.

switch {
case errors.Is(err, context.Canceled):
    slog.Debug("client disconnected", "path", r.URL.Path)
case err != nil:
    slog.Error("request failed", "err", err, "path", r.URL.Path)
}

Bridging Libraries That Use the Old log Package

Dependencies still writing to the standard log package do not have to break your JSON output:

// Route everything from the standard logger through slog at Info level.
slog.SetLogLoggerLevel(slog.LevelInfo)

// Or hand a specific component its own *log.Logger backed by slog.
srv := &http.Server{
    Addr:     ":8080",
    Handler:  mux,
    ErrorLog: slog.NewLogLogger(slog.Default().Handler(), slog.LevelError),
}

That second line is worth adding to every server you write — http.Server logs connection errors through ErrorLog, and by default they land on stderr as unstructured text that your aggregator will not parse.

Performance Notes

slog is designed so the common path allocates very little, but a few habits matter:

• Guard genuinely expensive debug work behind Enabled:

  if slog.Default().Enabled(ctx, slog.LevelDebug) {
      slog.Debug("payload", "body", expensiveDump(req))
  }
  

• Prefer LogAttrs in hot paths — it takes typed attributes and skips the any boxing:

  slog.LogAttrs(ctx, slog.LevelInfo, "cache hit",
      slog.String("key", key),
      slog.Int("size", len(val)),
  )
  

• Build the per-request logger once with With, not per log line.

None of this matters at ten requests per second. All of it matters in a hot loop — the same way caching with Ristretto only pays off once you are actually calling the expensive thing often.

Checklist

• One handler, configured once in main, installed with slog.SetDefault.
• JSON in production, text locally.
• A LevelVar so you can raise verbosity without a redeploy.
• A request-scoped logger in the context, carrying the request ID.
• ReplaceAttr or LogValuer for anything secret.
• Errors logged once, at the boundary, as values.
• http.Server.ErrorLog wired through slog.

Conclusion

log/slog removed the last dependency I used to add reflexively to every new Go service. It is fast enough, it is in the standard library, and its handler interface means you can change output format or destination without touching a single call site. If you are still logging with fmt.Sprintf, the migration is mostly mechanical — and the first time you filter a week of logs by request_id, you will not want to go back.

It pays off especially well for anything token-billed, where a handful of numeric fields per call is the only record of what a feature costs — calling an LLM from Go covers which ones to log.

The One Rule That Fixes Most of It

Handle an error once. Everywhere else, add context and pass it up.

“Handling” means doing something a caller cannot: returning a 404, retrying, falling back to a default, logging and moving on. If you are not doing one of those, you are just a link in the chain — and a link’s only job is to explain where it sits.

// Bad: the error is logged here and returned, so it will be logged again
// by every caller on the way up.
func (s *Store) User(ctx context.Context, id int64) (*User, error) {
    u, err := s.query(ctx, id)
    if err != nil {
        log.Printf("failed to load user: %v", err)
        return nil, err
    }
    return u, nil
}

// Good: add what this layer knows, and let the caller decide.
func (s *Store) User(ctx context.Context, id int64) (*User, error) {
    u, err := s.query(ctx, id)
    if err != nil {
        return nil, fmt.Errorf("load user %d: %w", id, err)
    }
    return u, nil
}

The second version produces messages that read like a stack trace written in English:

handle GET /users/42: load user 42: query users: sql: no rows in result set

Wrapping With %w

fmt.Errorf has a special verb, %w, that wraps the original error instead of flattening it to a string. The result behaves like a normal error, but the original stays reachable underneath.

_, err := os.Open("/etc/app/config.yaml")
wrapped := fmt.Errorf("read config: %w", err)

fmt.Println(wrapped)
// read config: open /etc/app/config.yaml: no such file or directory

fmt.Println(errors.Is(wrapped, os.ErrNotExist)) // true

That last line is the whole point. %v would have given you the same message, but errors.Is would have returned false because the chain was broken.

A few conventions that keep the output readable:

• Start the message with a lowercase verb phrase describing what this function was doing: "load user", "encode response", "dial redis".
• Do not end with punctuation, and do not include the word “error” or “failed” — the chain already reads as a failure.
• Put the wrapped error last, after a colon and a space.

Since Go 1.20 you can wrap more than one error in a single call, which is handy when you are cleaning up:

err := doWork()
if cerr := f.Close(); cerr != nil {
    err = fmt.Errorf("work failed: %w; close failed: %w", err, cerr)
}

Sentinel Errors: For Conditions Callers Branch On

A sentinel is a package-level error value that callers are meant to recognise.

package store

import "errors"

var (
    ErrNotFound  = errors.New("not found")
    ErrConflict  = errors.New("conflict")
    ErrForbidden = errors.New("forbidden")
)

Return them wrapped, so the caller gets both the identity and the context:

func (s *Store) User(ctx context.Context, id int64) (*User, error) {
    u, err := s.query(ctx, id)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, fmt.Errorf("load user %d: %w", id, ErrNotFound)
    }
    if err != nil {
        return nil, fmt.Errorf("load user %d: %w", id, err)
    }
    return u, nil
}

And check them with errors.Is, which walks the whole chain:

u, err := store.User(ctx, id)
switch {
case errors.Is(err, store.ErrNotFound):
    http.Error(w, "user not found", http.StatusNotFound)
    return
case err != nil:
    log.Error("load user", "err", err)
    http.Error(w, "internal error", http.StatusInternalServerError)
    return
}

Never compare with == when the error might be wrapped, and never compare error strings. errors.Is(err, ErrNotFound) keeps working when someone adds another wrapping layer three months from now; err == ErrNotFound silently stops working.

Keep the list short. Sentinels are part of your package’s public API — every one you export is a promise. If callers cannot meaningfully branch on it, it should not be a sentinel.

Custom Error Types: When You Need to Carry Data

A sentinel says what went wrong. A custom type also says with what. Reach for one when the caller needs a field, not just an identity.

// ValidationError reports a single field that failed validation.
type ValidationError struct {
    Field  string
    Reason string
}

func (e *ValidationError) Error() string {
    return fmt.Sprintf("field %q is invalid: %s", e.Field, e.Reason)
}

errors.As finds it anywhere in the chain and assigns it to your variable:

var vErr *ValidationError
if errors.As(err, &vErr) {
    w.WriteHeader(http.StatusBadRequest)
    json.NewEncoder(w).Encode(map[string]string{
        "field":  vErr.Field,
        "reason": vErr.Reason,
    })
    return
}

Two details that trip people up:

1. Pass a pointer to the target. errors.As(err, &vErr) where vErr is already a *ValidationError. Passing vErr directly panics.
2. Be consistent about pointer vs. value receivers. If Error() is defined on *ValidationError, then only *ValidationError implements error. Return &ValidationError{...}, and match against *ValidationError.

If your type wraps another error, give it an Unwrap method so errors.Is can keep walking:

type QueryError struct {
    Query string
    Err   error
}

func (e *QueryError) Error() string { return e.Query + ": " + e.Err.Error() }
func (e *QueryError) Unwrap() error { return e.Err }

Now errors.Is(err, sql.ErrNoRows) still works even with a QueryError in the middle.

Collecting Several Errors With errors.Join

When you validate a whole struct, failing on the first problem makes for a frustrating API. errors.Join (Go 1.20) combines errors into one value that errors.Is and errors.As can still search.

func (r CreateUserRequest) Validate() error {
    var errs []error
    if r.Email == "" {
        errs = append(errs, &ValidationError{Field: "email", Reason: "required"})
    }
    if len(r.Password) < 12 {
        errs = append(errs, &ValidationError{Field: "password", Reason: "too short"})
    }
    // Join returns nil when every element is nil, so this is safe as-is.
    return errors.Join(errs...)
}