How to Test Go Code That Calls an LLM

The first question everyone asks about a feature backed by a language model is “how do you even test that?” — usually with a shrug, as though non-determinism were a get-out-of-jail card for the whole test suite. It is not. Almost all of the code you write around a model is perfectly deterministic, and the small part that is not can be pinned down with a different kind of test. Here is how I split it.

Three Things, Tested Three Ways

The confusion comes from treating “the LLM feature” as one indivisible thing. It is three:

The first row is 90% of your code and needs no model at all. Get that boundary right and the rest is manageable.

Put an Interface in Front of the Model

Do not scatter client.Messages.New through your handlers. Define the narrowest interface your code actually needs and depend on that:

// Completer turns a prompt into text. It is deliberately narrow — narrower
// than the SDK — so tests can implement it in five lines.
type Completer interface {
    Complete(ctx context.Context, req Request) (Response, error)
}

type Request struct {
    System   string
    Messages []Message
    Tools    []Tool
}

type Response struct {
    Text       string
    ToolCalls  []ToolCall
    StopReason string
    Usage      Usage
}

This is just the dependency-inversion habit that makes any external service testable, and it earns its keep faster here than almost anywhere else. Two things fall out of it:

Your business logic never imports the SDK. It imports your Request and Response types. When the SDK’s union types change shape, one adapter file changes.

The fake is trivial. No HTTP, no fixtures, no mocking library:

type fakeCompleter struct {
    responses []Response
    err       error
    calls     []Request // recorded for assertions
}

func (f *fakeCompleter) Complete(_ context.Context, req Request) (Response, error) {
    f.calls = append(f.calls, req)
    if f.err != nil {
        return Response{}, f.err
    }
    if len(f.responses) == 0 {
        return Response{}, errors.New("fakeCompleter: no responses left")
    }
    resp := f.responses[0]
    f.responses = f.responses[1:]
    return resp, nil
}

Returning a queue rather than a single value is what lets you test multi-turn loops: the first call returns a tool request, the second returns the final answer.

Test the Loop, Not the Model

An agent loop has plenty of logic worth testing, none of which needs a model. Does it stop when it should? Does it return a result for every tool call? Does it hand a tool failure back rather than aborting?

func TestLoopReturnsResultForFailedTool(t *testing.T) {
    fake := &fakeCompleter{responses: []Response{
        {StopReason: "tool_use", ToolCalls: []ToolCall{
            {ID: "t1", Name: "lookup", Input: []byte(`{"id":"missing"}`)},
        }},
        {StopReason: "end_turn", Text: "I couldn't find that record."},
    }}

    agent := New(fake, map[string]ToolFunc{
        "lookup": func(context.Context, []byte) (string, error) {
            return "", errors.New("not found")
        },
    })

    got, err := agent.Run(context.Background(), "look up record missing")
    if err != nil {
        t.Fatalf("Run() error = %v, want nil", err)
    }
    if got != "I couldn't find that record." {
        t.Errorf("got %q", got)
    }

    // The failure must reach the model as a tool result, not abort the loop.
    second := fake.calls[1]
    result := findToolResult(t, second, "t1")
    if !result.IsError {
        t.Error("tool failure was not marked as an error result")
    }
}

That test catches a real bug — a loop that drops the failed call instead of reporting it — and runs in microseconds with no API key.

The same approach covers the rest of the loop’s contract — the invariants from the agent-loop article: that it stops at MaxIterations, that every tool_use block gets exactly one result, that results come back in one message. Table-driven tests fit this beautifully, since each case is just a different queue of canned responses.

Golden Files for Prompt Assembly

Prompt building is string manipulation, and it drifts. Someone adds a field, reorders a section, changes a heading — and unlike code, a prompt regression produces no compile error and no failing assertion, just slightly worse output that nobody attributes to the change.

Golden files make the diff visible in review:

var update = flag.Bool("update", false, "update golden files")

func TestBuildSystemPrompt(t *testing.T) {
    tests := []struct {
        name   string
        cfg    Config
        golden string
    }{
        {"default", Config{}, "system_default.txt"},
        {"with_tools", Config{Tools: allTools}, "system_with_tools.txt"},
        {"terse_mode", Config{Terse: true}, "system_terse.txt"},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := BuildSystemPrompt(tt.cfg)
            path := filepath.Join("testdata", tt.golden)

            if *update {
                if err := os.WriteFile(path, []byte(got), 0o644); err != nil {
                    t.Fatal(err)
                }
                return
            }

            want, err := os.ReadFile(path)
            if err != nil {
                t.Fatal(err)
            }
            if diff := cmp.Diff(string(want), got); diff != "" {
                t.Errorf("prompt changed (-want +got):\n%s", diff)
            }
        })
    }
}

Run go test ./... -update to accept a change deliberately. The point is not that the golden file is correct — it is that changing it requires saying so out loud, in a diff a reviewer can read.

This also catches the caching bugs from the prompt caching article before they cost you anything. A golden test on the system prompt fails the moment somebody interpolates time.Now() into it, because the output differs on every run.

Which suggests a second, blunter test worth having:

func TestPromptAssemblyIsDeterministic(t *testing.T) {
    // Go randomises map iteration order, so a prompt built by ranging over a
    // map differs run to run — and silently never caches.
    first := BuildSystemPrompt(cfg)
    for i := 0; i < 20; i++ {
        if got := BuildSystemPrompt(cfg); got != first {
            t.Fatalf("prompt is not deterministic on run %d", i)
        }
    }
}

Twenty iterations is enough to make a randomised map order fail essentially every time.

Integration Tests, Behind a Build Tag

You do need a handful of tests that touch the real API — enough to catch an SDK upgrade that changed a union type, a model id that no longer resolves, or streaming that broke. But they cost money and need a key, so they must not run on every go test ./...:

//go:build integration

package llm_test

func TestStreamingReturnsCompleteMessage(t *testing.T) {
    if os.Getenv("ANTHROPIC_API_KEY") == "" {
        t.Skip("no API key; skipping integration test")
    }
    // ... real call, assert on shape rather than content
}

go test ./...                    # fast, free, no key
go test -tags=integration ./...  # the real thing, on demand

Assert on shape, never on wording. resp.Text being non-empty, StopReason being end_turn, Usage.OutputTokens being greater than zero, a streamed message accumulating to the same content as a non-streamed one. Those hold across model versions. “The answer contains the word Paris” does not, and a flaky test that fails once a month teaches your team to ignore failures.

There is a middle option that gets you a long way for free: point the SDK at a local httptest.Server via the ANTHROPIC_BASE_URL environment variable and serve canned JSON. That exercises the real SDK — its parsing, its retry behaviour, its streaming decoder — without a key or a bill. It is the closest analogue to what go-sqlmock does for the database layer: a real driver, a fake server.

It is also the only sane way to test the paths you cannot easily provoke on purpose:

// Rate limiting: does the caller back off, or hammer?
mux.HandleFunc("/v1/messages", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Retry-After", "1")
    w.WriteHeader(http.StatusTooManyRequests)
    fmt.Fprint(w, `{"type":"error","error":{"type":"rate_limit_error","message":"..."}}`)
})

Do the same for a 500, a connection dropped mid-stream, and a malformed body. Those are the failures that actually page you, and they are trivial to simulate and nearly impossible to trigger on demand against the real API.

The Refusal Case Nobody Tests

Worth its own paragraph because it is so easy to miss: a safety refusal comes back as HTTP 200 with StopReason set to refusal. Your error handling never fires. Code that goes from if err != nil straight to reading the first content block treats it as an empty answer.

func TestRefusalIsNotTreatedAsEmptyAnswer(t *testing.T) {
    fake := &fakeCompleter{responses: []Response{
        {StopReason: "refusal", Text: ""},
    }}

    _, err := New(fake, nil).Run(context.Background(), "...")
    if !errors.Is(err, ErrDeclined) {
        t.Errorf("got %v, want ErrDeclined", err)
    }
}

One line in the fake, and you have covered a branch most production code does not have at all. The sentinel-error pattern behind ErrDeclined is the one from error handling in Go.

Evals: For Quality, Not Correctness

Everything above tests whether your code is right. None of it tells you whether the answers are any good. That needs a different instrument, and the mistake is trying to force it into go test.

An eval is a fixed set of inputs, run against the real model, scored rather than asserted:

//go:build eval

func TestClassificationAccuracy(t *testing.T) {
    cases := loadEvalCases(t, "testdata/eval/classification.jsonl")

    var correct int
    for _, c := range cases {
        got, err := classify(context.Background(), realClient, c.Input)
        if err != nil {
            t.Fatal(err)
        }
        if got == c.Want {
            correct++
        } else {
            t.Logf("MISS: input=%q got=%q want=%q", c.Input, got, c.Want)
        }
    }

    accuracy := float64(correct) / float64(len(cases))
    t.Logf("accuracy: %.1f%% (%d/%d)", accuracy*100, correct, len(cases))

    // A floor, not an equality check. Below this, something regressed.
    if accuracy < 0.90 {
        t.Errorf("accuracy %.1f%% below the 90%% floor", accuracy*100)
    }
}

Two things make this useful rather than annoying. The threshold is a floor, not a target — you are detecting regression, not demanding perfection. And the misses get logged, because the list of what it got wrong is the actual output; the pass/fail is almost incidental.

Run evals when you change a prompt, a model, or an effort setting — not on every commit. They cost money and take minutes.

What Not to Do

Do not assert on model wording. strings.Contains(resp, "Paris") passes today and fails after a model update that phrases it differently. It is not testing your code.

Do not set temperature to zero and call it deterministic. Sampling parameters are not even accepted on current models, and identical output was never guaranteed regardless.

Do not mock the SDK’s types. Mocking anthropic.Message and its union blocks is a lot of work to test the adapter you wrote to avoid exactly that. Fake your own interface instead.

Do not let integration tests run by default. A test suite that needs an API key is a test suite that new contributors cannot run, and CI cost that grows with every push.

Do not skip testing the error paths because they are “just the SDK’s job”. Rate limits, refusals and mid-stream disconnects are the failures you will actually see in production, and they are the cheapest things in this article to cover.

Checklist

• A narrow interface between your logic and the SDK; business code never imports the SDK.
• A fake with a queue of canned responses for multi-turn loops.
• Unit tests for the loop’s contract: one result per tool call, errors handed back, iteration cap honoured.
• Golden files for prompt assembly, plus a determinism test.
• Integration tests behind //go:build integration, asserting on shape not wording.
• An httptest.Server for 429s, 500s and truncated streams.
• A test for the refusal path.
• Evals behind their own tag, scored against a floor, misses logged.

Conclusion

“You cannot test LLM code” conflates the model with the code around it. The code around it — the prompt builder, the parser, the loop, the tool handlers, the error branches — is ordinary Go, and it becomes easy to test the moment there is an interface between it and the SDK. Push non-determinism out to the edges, cover the edges with evals scored against a floor, and the rest of your suite stays fast, free and green.