Calling an LLM from Go: Streaming, Timeouts and the Parts That Bite

Most of the LLM tutorials I read are Python notebooks. That is fine for a prototype, but the moment the thing has to live inside a service — with timeouts, cancellation, retries and a bill attached — Go’s constraints start to matter, and so do the details the notebooks skip. This is the write-up I wanted when I put my first Claude call into a Go HTTP handler.

Getting a Client

go get github.com/anthropics/anthropic-sdk-go

import (
    "github.com/anthropics/anthropic-sdk-go"
    "github.com/anthropics/anthropic-sdk-go/option"
)

// Reads ANTHROPIC_API_KEY from the environment.
client := anthropic.NewClient()

// Or pass it explicitly, if you load config yourself.
client := anthropic.NewClient(option.WithAPIKey(key))

Build the client once, at startup, and pass it around. It is safe for concurrent use and holds a connection pool — constructing one per request throws away keep-alive and gives you a fresh TLS handshake every time.

The First Call

resp, err := client.Messages.New(ctx, anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Messages: []anthropic.MessageParam{
        anthropic.NewUserMessage(anthropic.NewTextBlock("Summarise this changelog in three bullets.")),
    },
})
if err != nil {
    return fmt.Errorf("summarise changelog: %w", err)
}

Two things about that snippet are worth slowing down on.

The model is a plain string. The SDK ships typed constants (anthropic.ModelClaudeOpus4_8 and friends), but anthropic.Model is an alias for string, so a model that has no constant yet is passed as its id. Either form compiles; check the SDK release notes before assuming a constant exists for the model you want.

MaxTokens is a ceiling, not a target. It is the point at which generation is cut off mid-sentence, and the model is not told about it. Setting it to 500 because you want a short answer does not produce a short answer — it produces a truncated one. Ask for brevity in the prompt and leave the ceiling generous. For non-streaming requests, something around 16000 keeps you clear of both truncation and the SDK’s HTTP timeout.

Content Is a List of Blocks, Not a String

This is where the first hour usually goes. A response is not resp.Text. It is resp.Content, a slice of union values that can hold text, thinking, tool calls and more:

for _, block := range resp.Content {
    switch variant := block.AsAny().(type) {
    case anthropic.TextBlock:
        fmt.Println(variant.Text)
    case anthropic.ThinkingBlock:
        // Reasoning, when you have asked for it to be shown.
        log.Debug("model reasoning", "text", variant.Thinking)
    }
}

block.AsAny() is the accessor that gets you a concrete type to switch on. Reaching for resp.Content[0].Text and hoping works right up until the day a thinking block or a tool call lands in position zero, at which point you silently return an empty string. Write the type switch once, in a helper, and use it everywhere:

// firstText returns the first text block in a response, or "" if there is none.
func firstText(msg *anthropic.Message) string {
    for _, block := range msg.Content {
        if t, ok := block.AsAny().(anthropic.TextBlock); ok {
            return t.Text
        }
    }
    return ""
}

Thinking, and Why You Probably Want It On

Current Claude models can reason before answering. The recommended mode is adaptive — you do not budget tokens for it, the model decides how much thinking a given request deserves:

adaptive := anthropic.ThinkingConfigAdaptiveParam{}

params := anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Thinking:  anthropic.ThinkingConfigParamUnion{OfAdaptive: &adaptive},
    Messages:  messages,
}

There is no ThinkingConfigParamOfAdaptive helper — you construct the union literal and take the address of the variant, as above. That trips people up because almost every other option in the SDK does have a constructor function.

A word of warning if you are carrying settings over from older code: the fixed thinking budget (ThinkingConfigParamOfEnabled(N)) is gone on current models and returns a 400 rather than being ignored. If you want to spend less, the lever is effort, not a token budget — and effort lives inside output_config, not at the top level of the request.

The counter-intuitive part: on the newest models, turning thinking off is not reliably a cost saving. Lower effort with thinking on generally beats thinking off, and disabling it has failure modes of its own. Leave it on and turn effort down.

Streaming

Anything a user waits for should stream. It is not just perceived speed — a long non-streaming request is also the easiest way to hit an HTTP timeout.

stream := client.Messages.NewStreaming(ctx, anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 64000,
    Messages:  messages,
})

for stream.Next() {
    event := stream.Current()
    switch ev := event.AsAny().(type) {
    case anthropic.ContentBlockDeltaEvent:
        switch delta := ev.Delta.AsAny().(type) {
        case anthropic.TextDelta:
            fmt.Print(delta.Text)
        }
    }
}
if err := stream.Err(); err != nil {
    return fmt.Errorf("stream response: %w", err)
}

Two nested type switches is not the prettiest Go you will write, but the shape is stable: outer switch on the event, inner switch on the delta.

Always check stream.Err(). stream.Next() returning false means “no more events” — it does not tell you whether that was a clean finish or a dropped connection. A loop that ignores Err() will happily serve a truncated answer as if it were complete.

If you want the whole message and the incremental deltas, accumulate as you go. There is no GetFinalMessage() on the Go stream:

stream := client.Messages.NewStreaming(ctx, params)

var message anthropic.Message
for stream.Next() {
    message.Accumulate(stream.Current())
    // ... also forward the delta to the user here
}
if err := stream.Err(); err != nil {
    return err
}
// message.Content is now the complete response.

Raising MaxTokens to 64000 in the streaming example is deliberate. Timeouts stop being the binding constraint once you stream, so you can give the model room.

Streaming to a Browser

Server-sent events are the path of least resistance, and Go’s http.ResponseController makes the flushing straightforward:

func (h *Handler) stream(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("X-Accel-Buffering", "no") // stop nginx buffering the stream

    rc := http.NewResponseController(w)

    stream := h.client.Messages.NewStreaming(r.Context(), params)
    for stream.Next() {
        ev, ok := stream.Current().AsAny().(anthropic.ContentBlockDeltaEvent)
        if !ok {
            continue
        }
        delta, ok := ev.Delta.AsAny().(anthropic.TextDelta)
        if !ok {
            continue
        }

        // SSE data frames must not contain raw newlines.
        payload, _ := json.Marshal(delta.Text)
        if _, err := fmt.Fprintf(w, "data: %s\n\n", payload); err != nil {
            return // client hung up
        }
        if err := rc.Flush(); err != nil {
            return
        }
    }

    if err := stream.Err(); err != nil {
        slog.Error("llm stream failed", "err", err)
        fmt.Fprint(w, "event: error\ndata: {}\n\n")
        _ = rc.Flush()
        return
    }
    fmt.Fprint(w, "event: done\ndata: {}\n\n")
    _ = rc.Flush()
}

Three details that cost me an afternoon each:

• X-Accel-Buffering: no. Without it nginx buffers your stream and delivers the whole thing at the end, which looks exactly like streaming being broken. (Nginx has strong opinions about proxied responses generally — I ran into a related set of them in making Nginx cache cookie aware.)
• JSON-encode the delta. A model can and will emit a newline mid-sentence, and a bare newline terminates an SSE frame.
• Pass r.Context(), not context.Background(). When the user closes the tab, the request context cancels, the SDK aborts the HTTP call, and you stop paying for tokens nobody will read.

Deadlines and Cancellation

Context is not decoration here. It is the only thing standing between a slow model call and a goroutine that lives forever:

ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()

resp, err := client.Messages.New(ctx, params)

Set the deadline against how long the work should take, not a habit. A classification call has no business taking 90 seconds; a long agentic turn on a hard problem might legitimately run for several minutes. If you have not internalised how deadlines propagate through a call chain, understanding Golang context covers the machinery.

The corollary at shutdown: an in-flight model call is exactly the kind of long request that a naive SIGTERM handler will sever. Drain it properly — see graceful shutdown in Go web services.

Errors Worth Distinguishing

The SDK returns typed errors. Use errors.As to get at the status code rather than matching on message strings:

resp, err := client.Messages.New(ctx, params)
if err != nil {
    var apiErr *anthropic.Error
    if errors.As(err, &apiErr) {
        switch apiErr.StatusCode {
        case http.StatusTooManyRequests:
            return fmt.Errorf("rate limited: %w", ErrRetryable)
        case http.StatusBadRequest:
            // Your request is malformed. Retrying will not help.
            return fmt.Errorf("bad request to model: %w", err)
        default:
            return fmt.Errorf("model call failed (%d): %w", apiErr.StatusCode, err)
        }
    }
    // Not an API error: context cancellation, DNS, TLS, connection reset.
    return fmt.Errorf("model call failed: %w", err)
}

Wrapping with %w at each layer is what lets the HTTP boundary decide the status code without every layer needing to know about HTTP — the same pattern I laid out in error handling in Go.

The SDK already retries for you. By default it retries a couple of times on 408, 409, 429, 5xx and connection errors, with backoff. Two consequences people miss:

1. Do not add your own retry loop on top without lowering the SDK’s. Three of yours around two of its is nine attempts and a long tail of latency.
2. Wall-clock can reach timeout × (attempts + 1). Your context deadline is the real budget — set it deliberately, because the retry behaviour will happily use all of it.

A Refusal Is Not an Error

This one surprises people. If a safety classifier declines a request, you get HTTP 200 — a perfectly successful response whose StopReason says the model declined:

if resp.StopReason == anthropic.StopReasonRefusal {
    slog.Warn("model declined",
        "category", resp.StopDetails.Category,
        "explanation", resp.StopDetails.Explanation)
    return "", ErrDeclined
}

err is nil. resp.Content may hold nothing useful. Code that goes straight from if err != nil to reading Content[0] treats this as an empty answer and moves on. Check StopReason before you read content — and note the other values you care about: max_tokens means you were truncated, and tool_use means the model is waiting on you (which is a whole article of its own).

Watch the Usage Numbers

Every response carries a token accounting:

slog.Info("model call",
    "input_tokens", resp.Usage.InputTokens,
    "output_tokens", resp.Usage.OutputTokens,
    "cache_read", resp.Usage.CacheReadInputTokens,
    "cache_write", resp.Usage.CacheCreationInputTokens,
)

Log these from day one. They are the only ground truth about what a feature costs, and CacheReadInputTokens in particular is how you find out that your prompt caching silently stopped working three deploys ago — the failure mode there is not an error, just a bigger invoice. That is its own post, because it is the single biggest lever on what an LLM feature costs to run.

A structured logger pays for itself here: these are five numeric fields per call that you will want to aggregate later, which is exactly the case for structured logging with log/slog.

Checklist

• One client, built at startup, shared across handlers.
• Iterate resp.Content with a type switch; never index blindly into it.
• Adaptive thinking on; tune cost with effort, not by disabling it.
• Stream anything a human waits for, and always check stream.Err().
• r.Context() all the way down, with a deliberate deadline.
• errors.As into *anthropic.Error for status-code branching.
• Do not stack your retries on the SDK’s.
• Check StopReason before reading content — a refusal returns 200.
• Log the usage fields from the first commit.

Conclusion

The API surface is small — one endpoint, one loop, a handful of block types. What makes an LLM call different from any other HTTP call in your service is that it is slow, occasionally non-deterministic, priced per token, and able to succeed while declining to do what you asked. Go gives you good tools for exactly those problems, provided you use the context properly and read the response as the structured thing it is rather than the string you wish it were.