One Invariant, Everything Follows From It

Prompt caching is a prefix match. Any change anywhere in the prefix invalidates everything after it.

That is the whole model. The cache key is derived from the exact bytes of your rendered prompt up to each breakpoint. One byte different at position N — a timestamp, a reordered JSON key, an extra tool in the list — and every cached position at or after N is gone.

The render order matters and is fixed:

tools  →  system  →  messages

Tools render first, at position zero. That has a consequence people discover the expensive way: change the tool list and you have invalidated everything, system prompt and entire conversation included. More on that below.

What It Costs

Two numbers govern whether caching pays:

• A cache read costs about 0.1× the base input price.
• A cache write costs 1.25× for the 5-minute TTL, 2× for the 1-hour TTL.

So with the default 5-minute TTL, two requests already break even: 1.25 + 0.1 = 1.35 against 2.0 uncached. By the third request you are well ahead. With the 1-hour TTL you need three requests to break even, because the write costs double.

Which is why the TTL question is not “how long do I want this cached” but “how far apart do requests sharing this prefix start”:

The subtlety in row one: a read refreshes the entry at no extra cost, and the lifetime is measured from the start of the request. A four-minute generation leaves about one minute for the next request to begin before a five-minute entry expires. For a chat endpoint under steady load, the 5-minute TTL is the right answer and the 1-hour TTL just doubles your write bill.

Making It Work in Go

The syntax is a CacheControl on the last block of whatever you want cached. Because tools render before system, a marker on the final system block caches both:

params := anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Tools:     tools, // deterministic order — see below
    System: []anthropic.TextBlockParam{{
        Text:         systemPrompt,
        CacheControl: anthropic.NewCacheControlEphemeralParam(), // 5-minute default
    }},
    Messages: messages,
}

For the 1-hour TTL:

CacheControl: anthropic.CacheControlEphemeralParam{
    TTL: anthropic.CacheControlEphemeralTTLTTL1h,
},

There is also a top-level CacheControl on MessageNewParams that automatically places a breakpoint on the last cacheable block and moves it forward as the conversation grows. For multi-turn chat that is the right default — no marker bookkeeping, and the growing history caches incrementally.

The robust combination for anything agentic: one explicit breakpoint at the end of the static system prefix, so the expensive shared part has a guaranteed read point no matter what happens later in messages, plus top-level automatic caching for the growing tail.

You get four breakpoints per request, so there is no need to be frugal — place them at genuine stability boundaries.

Where Automatic Caching Is the Wrong Tool

Automatic placement puts the breakpoint at the very end of your prompt. When the prompt ends with something unique per request — a retrieved document, the user’s actual question — that is a pure surcharge: every request writes a new cache entry that nothing will ever read.

The signature is unmistakable once you know it: cache_creation_input_tokens is non-zero on every single request, while cache_read_input_tokens never covers the shared prefix.

The fix is an explicit marker at the end of the shared portion:

Messages: []anthropic.MessageParam{
    anthropic.NewUserMessage(
        anthropic.TextBlockParam{
            Text:         sharedContext, // few-shot examples, retrieved docs
            CacheControl: anthropic.NewCacheControlEphemeralParam(),
        },
        anthropic.NewTextBlock(userQuestion), // no marker — differs every time
    ),
},

Same rule, restated: put the breakpoint where the prompt stops being shared, not where the prompt ends.

The Minimum Prefix, Which Is Not Monotonic

A prompt shorter than the model’s minimum will not cache — no error, no warning, cache_creation_input_tokens simply comes back zero. And the minimum does not move in the direction you would guess as models get newer:

A 3,000-token system prompt caches on Opus 5 and Opus 4.8, and silently does not on Opus 4.6 or Haiku 4.5. If you switched models and your cache hit rate fell off a cliff, this is the first thing to check — and it cuts the other way too: moving to Opus 5 halves the Opus 4.8 minimum, so prompts that were previously too short start caching with no code change at all.

Silent Invalidators

This is the part worth committing to memory, because every one of these is code that looks perfectly reasonable in review.

The Go-specific ones deserve emphasis. Map iteration order in Go is deliberately randomised, so this is not a cache bug that appears under load — it appears on every request, and it is invisible because the rendered prompt is semantically identical each time:

// Bad: iteration order is randomised, so the bytes differ every run.
for name, def := range h.tools {
    tools = append(tools, def)
}

// Good: deterministic.
names := slices.Sorted(maps.Keys(h.tools))
for _, name := range names {
    tools = append(tools, h.tools[name])
}

Sorting keys is a one-line fix that most Go LLM code needs and almost none has.

Injecting Dynamic Context Without Breaking Everything

The usual reason a system prompt has a timestamp in it is that the model genuinely needs to know the date, or the user’s plan tier, or the current mode. The instinct is to template it into the system prompt. Don’t — that is the front of the prefix, and it invalidates everything behind it.

Put dynamic context after the cached history instead. On the newest models there is a first-class channel for this: a system-role message appended to messages, rather than an edit to the top-level system field.

// The top-level system prompt stays byte-identical and stays cached.
// The operator instruction goes after the history, invalidating nothing before it.
messages = append(messages, userTurn)
messages = append(messages, anthropic.MessageParam{
    Role: anthropic.MessageParamRoleSystem,
    Content: []anthropic.ContentBlockParamUnion{
        anthropic.NewTextBlock("Terse mode enabled — keep responses under 40 words."),
    },
})

A message at turn five invalidates nothing before turn five. That is the whole trick.

Two constraints: it must follow a user message and be either the last entry or followed by an assistant turn — it cannot be messages[0], so use the top-level system for the initial prompt. And support is model-dependent; unsupported models return a 400 saying the system role is not supported, so catch that and fall back to putting the instruction in a user turn.

Three Rules That Beat Marker Placement

Fix these before you fiddle with breakpoints.

Freeze the system prompt. No dates, no user names, no modes. It is the front of the prefix and everything downstream depends on it not moving.

Never change tools or model mid-conversation. Tools render at position zero, so adding, removing or reordering one invalidates the entire cache. Caches are also model-scoped, so switching models mid-conversation starts from cold. If you need “modes”, do not swap the tool set — pass the mode as message content.

Forked calls must reuse the parent’s exact prefix. Summarisation passes, sub-agents and side computations usually build their own request. If that fork rebuilds system, tools or model with any difference at all, it misses the parent’s cache completely. Copy them verbatim and append the fork-specific content at the end.

That last one is also the argument against a “cheap model for the easy stuff” cascade, at least as a first move. Caches are per model, so routing between two models forfeits cache reuse across them. Measure the capable model at lower effort before you build the cascade — it is often cheaper and simpler, and it keeps one cache namespace.

Verifying It, Forever

Every response carries the accounting (the same Usage struct I said to log from day one in calling an LLM from Go):

InputTokens is the uncached remainder only — not the prompt size. Total prompt = all three added together. If an agent ran for an hour and InputTokens reads 4K, the rest came from cache; check the sum, not the one field.

In a healthy multi-turn loop you should see, on each request:

• CacheReadInputTokens — the whole prior prefix, growing turn over turn.
• CacheCreationInputTokens — roughly the last assistant turn plus the new input. Small.
• InputTokens — just the tail past the last breakpoint.

If CacheCreationInputTokens is instead close to the full conversation size every time, the prefix is being rewritten upstream of your breakpoint. Go find the timestamp.

And then keep checking. The expensive failure here is never the bad first implementation — it is the regression. Caching works the day you write it, then six weeks later somebody adds a dynamic field to the system prompt or a tool list that stopped being sorted, and every request misses. Nothing fails. Nothing pages. You find out from finance.

So make it a standing assertion, not a one-time look:

func TestSystemPromptStaysCacheable(t *testing.T) {
    // Two identical requests: the second must read from cache.
    first := callModel(t, ctx, fixture)
    if first.Usage.CacheCreationInputTokens == 0 {
        t.Fatalf("nothing was cached: prompt may be under the model minimum")
    }

    second := callModel(t, ctx, fixture)
    if second.Usage.CacheReadInputTokens == 0 {
        t.Errorf("cache miss on an identical prompt — a silent invalidator crept in")
    }
}

That test costs a few cents to run and catches a regression that otherwise runs for months. Put it behind a build tag so it only runs when you mean it — the same treatment I gave integration tests in how to test database interactions in Golang. Better still, put a monitor on the ratio of CacheReadInputTokens to total input tokens and alert when it drops.

Checklist

• Frozen system prompt: no dates, IDs, names or conditional sections.
• Deterministic tool list, sorted by name, identical across users.
• One explicit breakpoint at the end of the static prefix; automatic caching for the tail.
• Breakpoint at the end of the shared portion, not the end of the prompt.
• 5-minute TTL under continuous traffic; 1-hour only for 5–60 minute gaps.
• Prompt above the model’s minimum, which changes between models.
• Dynamic context appended after the history, never templated into the system prompt.
• Forks copy the parent’s system, tools and model verbatim.
• A test or monitor on the usage fields, checked on every change to prompt assembly.

Conclusion

Prompt caching is the rare optimisation with no quality tradeoff: identical output, roughly a tenth of the input cost, and lower latency as a bonus. It is also unusually fragile, because it hinges on byte-exact prefixes and fails completely silently. Treat the prompt-building path the way you would treat a cache key anywhere else in your system — deterministic, stable, and covered by a test — and it mostly takes care of itself.

In our web application, we have multiple resolvers that execute some sql queries and return data in different types. Obviously, we want to prevent the database overloading by caching the same responses.

    // MyExampleType1 type to serve example response 1.
    type MyExampleType1 struct {}
    // MyExampleType2 type to serve example response 2.
    type MyExampleType2 struct {}

    // MyCachedResolver1 checks if there are any cached data by specific key.
    // If nothing in cache, queries the database and adds result to the redis cache.
    func MyCachedResolver1(redisClient *redis.Client) ([]MyExampleType1, error) {
      key := "resolver_result"
	  value, found := redisClient.Get(ctx, key)
	  if !found {
		value, err := FetchSomethingHeavyFromDB()
		if err != nil {
			return nil, err
		}

		valueJson, err := json.Marshal(value)
		if err != nil {
			return nil, err
		}
		redisClient.Set(context.Background(), key, valueJson, time.Hour*12)
		return value, nil
	  }

	  var val []MyExampleType1
	  err := json.Unmarshal([]byte(value.(string)), &val)
	  if err != nil {
		return nil, err
	  }
	  return val, nil
    }

    // MyCachedResolver2 checks if there are any cached data by specific key.
    // If nothing in cache, queries the database and adds result to the redis cache.
    func MyCachedResolver2(redisClient *redis.Client) ([]MyExampleType2, error) {
      key := "resolver_result_2"
	  value, found := redisClient.Get(ctx, key)
	  if !found {
		value, err := FetchSomethingEvenMoreHeavierFromDB()
		if err != nil {
			return nil, err
		}

		valueJson, err := json.Marshal(value)
		if err != nil {
			return nil, err
		}
		redisClient.Set(context.Background(), key, valueJson, time.Hour*12)
		return value, nil
	  }

	  var val []MyExampleType2
	  err := json.Unmarshal([]byte(value.(string)), &val)
	  if err != nil {
		return nil, err
	  }
	  return val, nil
    }

As you can see, there is some repetitive code that should be written due to the different types MyExampleType1 and MyExampleType2.

Now, let’s see how we can improve this situation by using generics. I’m going to write a function WithCache() which will be responsible for setting and getting data to/from redis cache.

    // WithCache adds exec function result into the redis cache.
    // Pay attention to "T any" that allows us to pass any type to this function.
    func WithCache[T any](redisClient *redis.Client, ctx context.Context, key string, exec func() (T, error), ttl time.Duration) (T, error) {
    	value, found := redisClient.Get(ctx, key)
    	var result T
    	if !found {
            // exec() is a function that should be executed to fetch needed data.
    		value, err := exec()
    		if err != nil {
    			return result, err
    		}
    		jsonValue, err := json.Marshal(value)
    		if err != nil {
    			return result, err
    		}
    		redisClient.Set(ctx, key, jsonValue, ttl)
    		return value, nil
    	}
    	var val T
    	err := json.Unmarshal([]byte(value.(string)), &val)
    	if err != nil {
    		return result, err
    	}
    	return val, nil
    }

After that our resolvers could be refactored into this:

     func MyCachedResolver1(redisClient *redis.Client) ([]MyExampleType1, error) {
       key := "resolver_result_1"
       return WithCache[[]MyExampleType1](redisClient, context.Background(), key, func() ([]MyExampleType1, error) {
		 return FetchSomethingHeavyFromDB()
	   }, time.Hour*12)
     }

     func MyCachedResolver2(redisClient *redis.Client) ([]MyExampleType2, error) {
       key := "resolver_result_2"
       return WithCache[[]MyExampleType2](redisClient, context.Background(), key, func() ([]MyExampleType2, error) {
		 return FetchSomethingEvenMoreHeavierFromDB()
	   }, time.Hour*12)
     }

Now it looks much better and clearer!

The caching code this example refactors is the one from Ristretto — the most performant concurrent cache library for Go, if you want the full context. For another place where a little type machinery removes a lot of duplication, see implementing enums in Golang. And since Go 1.27 a method can carry its own type parameters, which removes the package-level helper this article works around — what’s new in Go 1.27.

Ristretto is a fast, concurrent cache library built with a focus on performance and correctness.

This library was created by the Dgraph team as a contention-free cache for the Dgraph database.

Let’s dive into the practical example. We are going to build a simple application that gets a list of users from the database. In the first iteration, there will be no caching layer at all. In the second iteration, we will add a Ristretto caching and compare execution time.

Below, you can see that I defined a repository package with the Repository interface and with InMemoryRepository implementation:

package repository

import "fmt"

// Repository interface to handle users data.
type Repository interface {
	GetUsers() map[int]string
}

type InMemoryRepository struct{}

// NewInMemoryRepository constructs and returns InMemoryRepository.
func NewInMemoryRepository() *InMemoryRepository {
	return &InMemoryRepository{}
}

// GetUsers returns 50000 dummy users from the in-memory repository.
func (r *InMemoryRepository) GetUsers() map[int]string {
	users := make(map[int]string)
	for i:=1; i <= 50000; i++ {
		users[i] = fmt.Sprintf("User %d", i)
	}

	return users
}

Next, we are going to call a GetUsers() method 100 times to simulate calling of the same function from several places in the real-world applications:

package main

import (
	"fmt"
	"github.com/alexsergivan/blog-examples/ristretto/repository"
)

func main() {
	for i:=0; i<100; i ++ {
		UsersGetter(repository.NewInMemoryRepository())
	}
}

func UsersGetter(repository repository.Repository) {
	repository.GetUsers()
}

Let’s measure how much time it takes to execute it with time go run main.go:

1.46s user
0.34s system
106% cpu
1.686 total

Next, we are going to add a caching layer to our application.

Don’t forget to get the Ristretto library:

  go get github.com/dgraph-io/ristretto

Inside repository package we inject Ristretto cache:

package repository

import (
	"fmt"
	"github.com/dgraph-io/ristretto"
	"time"
)

type InMemoryRepository struct{
	cache        *ristretto.Cache
}

// NewInMemoryRepository constructs and returns InMemoryRepository.
func NewInMemoryRepository(ristrettoCache *ristretto.Cache) *InMemoryRepository {
	return &InMemoryRepository{
		cache: ristrettoCache,
	}
}

// GetUsers returns 50000 dummy users from the in-memory repository.
func (r *InMemoryRepository) GetUsers() map[int]string {
	key := "users"
	value, found := r.cache.Get(key)
    // If the users data not cached yet, get it from the repository.
	if !found {
		users := make(map[int]string)
		for i:=1; i <= 50000; i++ {
			users[i] = fmt.Sprintf("User %d", i)
		}
        // Adds data to the cache for 1h.
		r.cache.SetWithTTL(key, users, 1, 1*time.Hour)
		time.Sleep(10 * time.Millisecond)
		return users
	}
	return value.(map[int]string)
}

Next, inside the main() function we initiate a new Ristretto cache and pass it to the InMemoryRepository:

  package main

import (
	"github.com/alexsergivan/blog-examples/ristretto/repository"
	"github.com/dgraph-io/ristretto"
)

func main() {
	ristrettoCache, _ := ristretto.NewCache(&ristretto.Config{
		NumCounters: 1e7,     // Num keys to track frequency of (10M).
		MaxCost:     1 << 30, // Maximum cost of cache (1GB).
		BufferItems: 64,      // Number of keys per Get buffer.
	})

	for i:=0; i<100; i ++ {
		UsersGetter(repository.NewInMemoryRepository(ristrettoCache))
	}
}

func UsersGetter(repository repository.Repository) {
	repository.GetUsers()
}

Let’s check how much time it takes to perform the same action:

0.29s user
0.26s system
147% cpu
0.377 total

As you can notice, the total time is 4 times less than in the example without caching layer.

Despite a silly example, I hope you got an idea of how to integrate the Ristretto caching into your application and how it could improve overall performance.

The complete source code you can find here.

Ristretto is safe for concurrent use, which is exactly the problem a plain map does not solve — see concurrent map writing and reading in Go if you want the failure mode in detail. I also refactored this caching layer with generics in how Golang generics minimize the amount of code you need to write. For a cache with a very different failure mode — one that costs you money rather than latency when it silently stops working — see prompt caching.

Let’s imagine a situation when you want to do an A/B test on your website, where 50% of users should see new headline text on the page. Other 50% of visitors will continue see the old page. In this case, all your server-side manipulation about splitting users to different versions will be ignored by nginx cache (of course, if you have it).

It happens because nginx, by default, has this configuration:

proxy_cache_key $scheme$proxy_host$request_uri;

So, it will use the same cache key for all users, who requests a page with the same url.

Luckily, nginx allows us easily to customize proxy_cache_key! What we need to do, it’s just to add a specific cookie to this key:

proxy_cache_key $scheme$proxy_host$request_uri$cookie_MY_COOKIE_NAME;

And that’s it! After this, if userA has MY_COOKIE_NAME=A and userB has MY_COOKIE_NAME=B they will receive different versions of page by the same url.

If you need more complex behaviour, where you won’t use hardcoded cookie name in your nginx config, you can do something like this:

if ($http_cookie ~* "ab_(.*?)=([\w-]+)" ) {
  set $abcookie $1$2;
}

proxy_cache_key $scheme$proxy_host$request_uri$abcookie; 

As you can see, you can use $http_cookie and generate proxy_cache_key based on specific cookie patterns. In this concrete example we check if http cookies contains cookie with regex pattern ab_(.*?)=([\w-]+) and if this cookie exists, we generate new variable for proxy_cache_key`.

Caching at the proxy is only one layer. For the one inside your Go process, see Ristretto — the most performant concurrent cache library for Go, and for shaving latency off the browser side, 103 Early Hints in Go.