Prompt Caching: The Cheapest Win in Your LLM Bill

I have written on this blog about caching database reads with Ristretto and about teaching Nginx to cache by cookie. Prompt caching belongs in the same family, with one difference that makes it far more interesting: the thing you are caching costs real money per byte, and when the cache stops working, nothing breaks. No error, no alert, no failed request. Just a bigger invoice next month.

One Invariant, Everything Follows From It

Prompt caching is a prefix match. Any change anywhere in the prefix invalidates everything after it.

That is the whole model. The cache key is derived from the exact bytes of your rendered prompt up to each breakpoint. One byte different at position N — a timestamp, a reordered JSON key, an extra tool in the list — and every cached position at or after N is gone.

The render order matters and is fixed:

tools  →  system  →  messages

Tools render first, at position zero. That has a consequence people discover the expensive way: change the tool list and you have invalidated everything, system prompt and entire conversation included. More on that below.

What It Costs

Two numbers govern whether caching pays:

• A cache read costs about 0.1× the base input price.
• A cache write costs 1.25× for the 5-minute TTL, 2× for the 1-hour TTL.

So with the default 5-minute TTL, two requests already break even: 1.25 + 0.1 = 1.35 against 2.0 uncached. By the third request you are well ahead. With the 1-hour TTL you need three requests to break even, because the write costs double.

Which is why the TTL question is not “how long do I want this cached” but “how far apart do requests sharing this prefix start”:

The subtlety in row one: a read refreshes the entry at no extra cost, and the lifetime is measured from the start of the request. A four-minute generation leaves about one minute for the next request to begin before a five-minute entry expires. For a chat endpoint under steady load, the 5-minute TTL is the right answer and the 1-hour TTL just doubles your write bill.

Making It Work in Go

The syntax is a CacheControl on the last block of whatever you want cached. Because tools render before system, a marker on the final system block caches both:

params := anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Tools:     tools, // deterministic order — see below
    System: []anthropic.TextBlockParam{{
        Text:         systemPrompt,
        CacheControl: anthropic.NewCacheControlEphemeralParam(), // 5-minute default
    }},
    Messages: messages,
}

For the 1-hour TTL:

CacheControl: anthropic.CacheControlEphemeralParam{
    TTL: anthropic.CacheControlEphemeralTTLTTL1h,
},

There is also a top-level CacheControl on MessageNewParams that automatically places a breakpoint on the last cacheable block and moves it forward as the conversation grows. For multi-turn chat that is the right default — no marker bookkeeping, and the growing history caches incrementally.

The robust combination for anything agentic: one explicit breakpoint at the end of the static system prefix, so the expensive shared part has a guaranteed read point no matter what happens later in messages, plus top-level automatic caching for the growing tail.

You get four breakpoints per request, so there is no need to be frugal — place them at genuine stability boundaries.

Where Automatic Caching Is the Wrong Tool

Automatic placement puts the breakpoint at the very end of your prompt. When the prompt ends with something unique per request — a retrieved document, the user’s actual question — that is a pure surcharge: every request writes a new cache entry that nothing will ever read.

The signature is unmistakable once you know it: cache_creation_input_tokens is non-zero on every single request, while cache_read_input_tokens never covers the shared prefix.

The fix is an explicit marker at the end of the shared portion:

Messages: []anthropic.MessageParam{
    anthropic.NewUserMessage(
        anthropic.TextBlockParam{
            Text:         sharedContext, // few-shot examples, retrieved docs
            CacheControl: anthropic.NewCacheControlEphemeralParam(),
        },
        anthropic.NewTextBlock(userQuestion), // no marker — differs every time
    ),
},

Same rule, restated: put the breakpoint where the prompt stops being shared, not where the prompt ends.

The Minimum Prefix, Which Is Not Monotonic

A prompt shorter than the model’s minimum will not cache — no error, no warning, cache_creation_input_tokens simply comes back zero. And the minimum does not move in the direction you would guess as models get newer:

A 3,000-token system prompt caches on Opus 5 and Opus 4.8, and silently does not on Opus 4.6 or Haiku 4.5. If you switched models and your cache hit rate fell off a cliff, this is the first thing to check — and it cuts the other way too: moving to Opus 5 halves the Opus 4.8 minimum, so prompts that were previously too short start caching with no code change at all.

Silent Invalidators

This is the part worth committing to memory, because every one of these is code that looks perfectly reasonable in review.

The Go-specific ones deserve emphasis. Map iteration order in Go is deliberately randomised, so this is not a cache bug that appears under load — it appears on every request, and it is invisible because the rendered prompt is semantically identical each time:

// Bad: iteration order is randomised, so the bytes differ every run.
for name, def := range h.tools {
    tools = append(tools, def)
}

// Good: deterministic.
names := slices.Sorted(maps.Keys(h.tools))
for _, name := range names {
    tools = append(tools, h.tools[name])
}

Sorting keys is a one-line fix that most Go LLM code needs and almost none has.

Injecting Dynamic Context Without Breaking Everything

The usual reason a system prompt has a timestamp in it is that the model genuinely needs to know the date, or the user’s plan tier, or the current mode. The instinct is to template it into the system prompt. Don’t — that is the front of the prefix, and it invalidates everything behind it.

Put dynamic context after the cached history instead. On the newest models there is a first-class channel for this: a system-role message appended to messages, rather than an edit to the top-level system field.

// The top-level system prompt stays byte-identical and stays cached.
// The operator instruction goes after the history, invalidating nothing before it.
messages = append(messages, userTurn)
messages = append(messages, anthropic.MessageParam{
    Role: anthropic.MessageParamRoleSystem,
    Content: []anthropic.ContentBlockParamUnion{
        anthropic.NewTextBlock("Terse mode enabled — keep responses under 40 words."),
    },
})

A message at turn five invalidates nothing before turn five. That is the whole trick.

Two constraints: it must follow a user message and be either the last entry or followed by an assistant turn — it cannot be messages[0], so use the top-level system for the initial prompt. And support is model-dependent; unsupported models return a 400 saying the system role is not supported, so catch that and fall back to putting the instruction in a user turn.

Three Rules That Beat Marker Placement

Fix these before you fiddle with breakpoints.

Freeze the system prompt. No dates, no user names, no modes. It is the front of the prefix and everything downstream depends on it not moving.

Never change tools or model mid-conversation. Tools render at position zero, so adding, removing or reordering one invalidates the entire cache. Caches are also model-scoped, so switching models mid-conversation starts from cold. If you need “modes”, do not swap the tool set — pass the mode as message content.

Forked calls must reuse the parent’s exact prefix. Summarisation passes, sub-agents and side computations usually build their own request. If that fork rebuilds system, tools or model with any difference at all, it misses the parent’s cache completely. Copy them verbatim and append the fork-specific content at the end.

That last one is also the argument against a “cheap model for the easy stuff” cascade, at least as a first move. Caches are per model, so routing between two models forfeits cache reuse across them. Measure the capable model at lower effort before you build the cascade — it is often cheaper and simpler, and it keeps one cache namespace.

Verifying It, Forever

Every response carries the accounting (the same Usage struct I said to log from day one in calling an LLM from Go):

InputTokens is the uncached remainder only — not the prompt size. Total prompt = all three added together. If an agent ran for an hour and InputTokens reads 4K, the rest came from cache; check the sum, not the one field.

In a healthy multi-turn loop you should see, on each request:

• CacheReadInputTokens — the whole prior prefix, growing turn over turn.
• CacheCreationInputTokens — roughly the last assistant turn plus the new input. Small.
• InputTokens — just the tail past the last breakpoint.

If CacheCreationInputTokens is instead close to the full conversation size every time, the prefix is being rewritten upstream of your breakpoint. Go find the timestamp.

And then keep checking. The expensive failure here is never the bad first implementation — it is the regression. Caching works the day you write it, then six weeks later somebody adds a dynamic field to the system prompt or a tool list that stopped being sorted, and every request misses. Nothing fails. Nothing pages. You find out from finance.

So make it a standing assertion, not a one-time look:

func TestSystemPromptStaysCacheable(t *testing.T) {
    // Two identical requests: the second must read from cache.
    first := callModel(t, ctx, fixture)
    if first.Usage.CacheCreationInputTokens == 0 {
        t.Fatalf("nothing was cached: prompt may be under the model minimum")
    }

    second := callModel(t, ctx, fixture)
    if second.Usage.CacheReadInputTokens == 0 {
        t.Errorf("cache miss on an identical prompt — a silent invalidator crept in")
    }
}

That test costs a few cents to run and catches a regression that otherwise runs for months. Put it behind a build tag so it only runs when you mean it — the same treatment I gave integration tests in how to test database interactions in Golang. Better still, put a monitor on the ratio of CacheReadInputTokens to total input tokens and alert when it drops.

Checklist

• Frozen system prompt: no dates, IDs, names or conditional sections.
• Deterministic tool list, sorted by name, identical across users.
• One explicit breakpoint at the end of the static prefix; automatic caching for the tail.
• Breakpoint at the end of the shared portion, not the end of the prompt.
• 5-minute TTL under continuous traffic; 1-hour only for 5–60 minute gaps.
• Prompt above the model’s minimum, which changes between models.
• Dynamic context appended after the history, never templated into the system prompt.
• Forks copy the parent’s system, tools and model verbatim.
• A test or monitor on the usage fields, checked on every change to prompt assembly.

Conclusion

Prompt caching is the rare optimisation with no quality tradeoff: identical output, roughly a tenth of the input cost, and lower latency as a bonus. It is also unusually fragile, because it hinges on byte-exact prefixes and fails completely silently. Treat the prompt-building path the way you would treat a cache key anywhere else in your system — deterministic, stable, and covered by a test — and it mostly takes care of itself.