One Invariant, Everything Follows From It

Prompt caching is a prefix match. Any change anywhere in the prefix invalidates everything after it.

That is the whole model. The cache key is derived from the exact bytes of your rendered prompt up to each breakpoint. One byte different at position N — a timestamp, a reordered JSON key, an extra tool in the list — and every cached position at or after N is gone.

The render order matters and is fixed:

tools  →  system  →  messages

Tools render first, at position zero. That has a consequence people discover the expensive way: change the tool list and you have invalidated everything, system prompt and entire conversation included. More on that below.

What It Costs

Two numbers govern whether caching pays:

• A cache read costs about 0.1× the base input price.
• A cache write costs 1.25× for the 5-minute TTL, 2× for the 1-hour TTL.

So with the default 5-minute TTL, two requests already break even: 1.25 + 0.1 = 1.35 against 2.0 uncached. By the third request you are well ahead. With the 1-hour TTL you need three requests to break even, because the write costs double.

Which is why the TTL question is not “how long do I want this cached” but “how far apart do requests sharing this prefix start”:

The subtlety in row one: a read refreshes the entry at no extra cost, and the lifetime is measured from the start of the request. A four-minute generation leaves about one minute for the next request to begin before a five-minute entry expires. For a chat endpoint under steady load, the 5-minute TTL is the right answer and the 1-hour TTL just doubles your write bill.

Making It Work in Go

The syntax is a CacheControl on the last block of whatever you want cached. Because tools render before system, a marker on the final system block caches both:

params := anthropic.MessageNewParams{
    Model:     "claude-opus-5",
    MaxTokens: 16000,
    Tools:     tools, // deterministic order — see below
    System: []anthropic.TextBlockParam{{
        Text:         systemPrompt,
        CacheControl: anthropic.NewCacheControlEphemeralParam(), // 5-minute default
    }},
    Messages: messages,
}

For the 1-hour TTL:

CacheControl: anthropic.CacheControlEphemeralParam{
    TTL: anthropic.CacheControlEphemeralTTLTTL1h,
},

There is also a top-level CacheControl on MessageNewParams that automatically places a breakpoint on the last cacheable block and moves it forward as the conversation grows. For multi-turn chat that is the right default — no marker bookkeeping, and the growing history caches incrementally.

The robust combination for anything agentic: one explicit breakpoint at the end of the static system prefix, so the expensive shared part has a guaranteed read point no matter what happens later in messages, plus top-level automatic caching for the growing tail.

You get four breakpoints per request, so there is no need to be frugal — place them at genuine stability boundaries.

Where Automatic Caching Is the Wrong Tool

Automatic placement puts the breakpoint at the very end of your prompt. When the prompt ends with something unique per request — a retrieved document, the user’s actual question — that is a pure surcharge: every request writes a new cache entry that nothing will ever read.

The signature is unmistakable once you know it: cache_creation_input_tokens is non-zero on every single request, while cache_read_input_tokens never covers the shared prefix.

The fix is an explicit marker at the end of the shared portion:

Messages: []anthropic.MessageParam{
    anthropic.NewUserMessage(
        anthropic.TextBlockParam{
            Text:         sharedContext, // few-shot examples, retrieved docs
            CacheControl: anthropic.NewCacheControlEphemeralParam(),
        },
        anthropic.NewTextBlock(userQuestion), // no marker — differs every time
    ),
},

Same rule, restated: put the breakpoint where the prompt stops being shared, not where the prompt ends.

The Minimum Prefix, Which Is Not Monotonic

A prompt shorter than the model’s minimum will not cache — no error, no warning, cache_creation_input_tokens simply comes back zero. And the minimum does not move in the direction you would guess as models get newer:

A 3,000-token system prompt caches on Opus 5 and Opus 4.8, and silently does not on Opus 4.6 or Haiku 4.5. If you switched models and your cache hit rate fell off a cliff, this is the first thing to check — and it cuts the other way too: moving to Opus 5 halves the Opus 4.8 minimum, so prompts that were previously too short start caching with no code change at all.

Silent Invalidators

This is the part worth committing to memory, because every one of these is code that looks perfectly reasonable in review.

The Go-specific ones deserve emphasis. Map iteration order in Go is deliberately randomised, so this is not a cache bug that appears under load — it appears on every request, and it is invisible because the rendered prompt is semantically identical each time:

// Bad: iteration order is randomised, so the bytes differ every run.
for name, def := range h.tools {
    tools = append(tools, def)
}

// Good: deterministic.
names := slices.Sorted(maps.Keys(h.tools))
for _, name := range names {
    tools = append(tools, h.tools[name])
}

Sorting keys is a one-line fix that most Go LLM code needs and almost none has.

Injecting Dynamic Context Without Breaking Everything

The usual reason a system prompt has a timestamp in it is that the model genuinely needs to know the date, or the user’s plan tier, or the current mode. The instinct is to template it into the system prompt. Don’t — that is the front of the prefix, and it invalidates everything behind it.

Put dynamic context after the cached history instead. On the newest models there is a first-class channel for this: a system-role message appended to messages, rather than an edit to the top-level system field.

// The top-level system prompt stays byte-identical and stays cached.
// The operator instruction goes after the history, invalidating nothing before it.
messages = append(messages, userTurn)
messages = append(messages, anthropic.MessageParam{
    Role: anthropic.MessageParamRoleSystem,
    Content: []anthropic.ContentBlockParamUnion{
        anthropic.NewTextBlock("Terse mode enabled — keep responses under 40 words."),
    },
})

A message at turn five invalidates nothing before turn five. That is the whole trick.

Two constraints: it must follow a user message and be either the last entry or followed by an assistant turn — it cannot be messages[0], so use the top-level system for the initial prompt. And support is model-dependent; unsupported models return a 400 saying the system role is not supported, so catch that and fall back to putting the instruction in a user turn.

Three Rules That Beat Marker Placement

Fix these before you fiddle with breakpoints.

Freeze the system prompt. No dates, no user names, no modes. It is the front of the prefix and everything downstream depends on it not moving.

Never change tools or model mid-conversation. Tools render at position zero, so adding, removing or reordering one invalidates the entire cache. Caches are also model-scoped, so switching models mid-conversation starts from cold. If you need “modes”, do not swap the tool set — pass the mode as message content.

Forked calls must reuse the parent’s exact prefix. Summarisation passes, sub-agents and side computations usually build their own request. If that fork rebuilds system, tools or model with any difference at all, it misses the parent’s cache completely. Copy them verbatim and append the fork-specific content at the end.

That last one is also the argument against a “cheap model for the easy stuff” cascade, at least as a first move. Caches are per model, so routing between two models forfeits cache reuse across them. Measure the capable model at lower effort before you build the cascade — it is often cheaper and simpler, and it keeps one cache namespace.

Verifying It, Forever

Every response carries the accounting (the same Usage struct I said to log from day one in calling an LLM from Go):

InputTokens is the uncached remainder only — not the prompt size. Total prompt = all three added together. If an agent ran for an hour and InputTokens reads 4K, the rest came from cache; check the sum, not the one field.

In a healthy multi-turn loop you should see, on each request:

• CacheReadInputTokens — the whole prior prefix, growing turn over turn.
• CacheCreationInputTokens — roughly the last assistant turn plus the new input. Small.
• InputTokens — just the tail past the last breakpoint.

If CacheCreationInputTokens is instead close to the full conversation size every time, the prefix is being rewritten upstream of your breakpoint. Go find the timestamp.

And then keep checking. The expensive failure here is never the bad first implementation — it is the regression. Caching works the day you write it, then six weeks later somebody adds a dynamic field to the system prompt or a tool list that stopped being sorted, and every request misses. Nothing fails. Nothing pages. You find out from finance.

So make it a standing assertion, not a one-time look:

func TestSystemPromptStaysCacheable(t *testing.T) {
    // Two identical requests: the second must read from cache.
    first := callModel(t, ctx, fixture)
    if first.Usage.CacheCreationInputTokens == 0 {
        t.Fatalf("nothing was cached: prompt may be under the model minimum")
    }

    second := callModel(t, ctx, fixture)
    if second.Usage.CacheReadInputTokens == 0 {
        t.Errorf("cache miss on an identical prompt — a silent invalidator crept in")
    }
}

That test costs a few cents to run and catches a regression that otherwise runs for months. Put it behind a build tag so it only runs when you mean it — the same treatment I gave integration tests in how to test database interactions in Golang. Better still, put a monitor on the ratio of CacheReadInputTokens to total input tokens and alert when it drops.

Checklist

• Frozen system prompt: no dates, IDs, names or conditional sections.
• Deterministic tool list, sorted by name, identical across users.
• One explicit breakpoint at the end of the static prefix; automatic caching for the tail.
• Breakpoint at the end of the shared portion, not the end of the prompt.
• 5-minute TTL under continuous traffic; 1-hour only for 5–60 minute gaps.
• Prompt above the model’s minimum, which changes between models.
• Dynamic context appended after the history, never templated into the system prompt.
• Forks copy the parent’s system, tools and model verbatim.
• A test or monitor on the usage fields, checked on every change to prompt assembly.

Conclusion

Prompt caching is the rare optimisation with no quality tradeoff: identical output, roughly a tenth of the input cost, and lower latency as a bonus. It is also unusually fragile, because it hinges on byte-exact prefixes and fails completely silently. Treat the prompt-building path the way you would treat a cache key anywhere else in your system — deterministic, stable, and covered by a test — and it mostly takes care of itself.

The Problem With the Obvious Version

// Do not ship this.
func fetchAll(urls []string) {
    var wg sync.WaitGroup
    for _, url := range urls {
        wg.Add(1)
        go func() {
            defer wg.Done()
            fetch(url)
        }()
    }
    wg.Wait()
}

Three separate problems:

1. No limit. len(urls) concurrent requests. The remote service rate-limits you, or your file descriptors run out, or both.
2. No errors. fetch returns one and it goes nowhere.
3. No cancellation. If the caller gives up, every goroutine keeps running to completion.

The concurrency itself is not the mistake — the missing back pressure is.

The Classic Channel Pool

The traditional shape is a jobs channel, a fixed number of workers reading from it, and a results channel:

type job struct {
    ID  int
    URL string
}

type result struct {
    JobID int
    Body  []byte
    Err   error
}

func workerPool(ctx context.Context, jobs []job, workers int) []result {
    jobCh := make(chan job)
    resCh := make(chan result, len(jobs))

    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func(workerID int) {
            defer wg.Done()
            for j := range jobCh {
                body, err := fetch(ctx, j.URL)
                select {
                case resCh <- result{JobID: j.ID, Body: body, Err: err}:
                case <-ctx.Done():
                    return
                }
            }
        }(i)
    }

    // Feed the workers, stopping early if the caller cancels.
    go func() {
        defer close(jobCh)
        for _, j := range jobs {
            select {
            case jobCh <- j:
            case <-ctx.Done():
                return
            }
        }
    }()

    wg.Wait()
    close(resCh)

    out := make([]result, 0, len(jobs))
    for r := range resCh {
        out = append(out, r)
    }
    return out
}

This is worth understanding because you will read it in a lot of codebases, and because it shows the mechanics plainly. Note two things that are easy to get wrong:

• close(jobCh) is the workers’ exit signal. for j := range jobCh ends when the channel closes. Forget the close and wg.Wait() blocks forever.
• Every channel send is paired with <-ctx.Done(). Without that, a worker sending to a full resCh that nobody is reading leaks for the lifetime of the process.

It is also about forty lines to do something the standard extended library does in eight.

The errgroup Version

golang.org/x/sync/errgroup is a sync.WaitGroup that also collects the first error and cancels its siblings. SetLimit turns it into a bounded pool:

import "golang.org/x/sync/errgroup"

func fetchAll(ctx context.Context, urls []string) ([][]byte, error) {
    bodies := make([][]byte, len(urls))

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(10) // at most 10 in flight

    for i, url := range urls {
        g.Go(func() error {
            body, err := fetch(ctx, url)
            if err != nil {
                return fmt.Errorf("fetch %s: %w", url, err)
            }
            // Each goroutine owns exactly one slot: no mutex needed.
            bodies[i] = body
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }
    return bodies, nil
}

That is the whole pool. Behaviour worth knowing:

• g.Go blocks once the limit is reached, until a slot frees up. The for loop becomes its own back pressure — no jobs channel needed.
• errgroup.WithContext returns a derived context that is cancelled the moment any goroutine returns a non-nil error. Shadowing ctx with it, as above, is deliberate: every fetch gets the cancellable one.
• g.Wait() returns the first error, and waits for the rest regardless. Later errors are discarded — if you need all of them, collect them yourself (errors.Join is a good fit, see error handling in Go).
• Writing to bodies[i] is safe without a mutex because each goroutine writes one distinct element. Different elements of a slice are different memory; that is not a data race. Appending to a shared slice, or writing to a shared map, absolutely is — see concurrent map writing and reading in Go for what that failure looks like.

A Note on Loop Variables

The example above relies on Go 1.22’s per-iteration loop variables. Before 1.22, i and url were shared across iterations and every goroutine would see the final values — the single most common concurrency bug in Go. On older versions you must copy them:

for i, url := range urls {
    i, url := i, url // required before Go 1.22
    g.Go(func() error { /* ... */ })
}

Since Go 1.22 the copy is unnecessary. Leaving it in is harmless, and I still write it in code that must build on older toolchains. The loop semantics change was one of the more consequential recent additions to the language — I touched on the surrounding rules in mastering Golang for loops.

Streaming Results Instead of Preallocating

Indexing into a preallocated slice only works when you know the number of jobs up front. For a stream, send results down a channel and read them concurrently:

func processStream(ctx context.Context, in <-chan job, workers int) (<-chan result, func() error) {
    out := make(chan result)

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(workers)

    go func() {
        // Closing `out` after every worker has finished lets the consumer
        // range over it and stop naturally.
        defer close(out)

        for j := range in {
            g.Go(func() error {
                body, err := fetch(ctx, j.URL)
                if err != nil {
                    return fmt.Errorf("job %d: %w", j.ID, err)
                }
                select {
                case out <- result{JobID: j.ID, Body: body}:
                    return nil
                case <-ctx.Done():
                    return ctx.Err()
                }
            })
        }
        _ = g.Wait()
    }()

    return out, g.Wait
}

The consumer ranges over out and then calls the returned function to get the error:

results, wait := processStream(ctx, jobs, 8)
for r := range results {
    save(r)
}
if err := wait(); err != nil {
    return fmt.Errorf("process stream: %w", err)
}

g.Wait() is safe to call more than once — subsequent calls return the same error immediately.

Picking the Limit

There is no universal number, but there is a reliable way to think about it.

CPU-bound work — parsing, hashing, image resizing, compression — saturates at roughly the number of cores. More goroutines just add scheduling overhead:

g.SetLimit(runtime.GOMAXPROCS(0))

I/O-bound work — HTTP calls, database queries, object storage — spends most of its time waiting, so the useful limit is much higher. But it is not “as high as possible”: it is whatever the slowest downstream dependency can absorb. If your database pool has 25 connections, a pool of 200 workers means 175 goroutines queueing on a mutex inside database/sql while your latency graph climbs.

// Match the constraint that actually binds.
g.SetLimit(db.Stats().MaxOpenConnections)

For outbound HTTP, remember that Go’s default transport keeps only 2 idle connections per host. Exceed that and you are opening a fresh TCP connection — plus a TLS handshake — for each extra request:

transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConnsPerHost = 50
transport.MaxConnsPerHost = 50
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

Then set the pool limit to match. Tuning one without the other gets you nothing.

Whatever you pick, measure it. Run the job at 5, 10, 25 and 50 and look at total wall time and downstream latency — the fastest setting for your batch is often the one that makes everything else on the system slower. Load testing is the honest way to find out; I wrote about a lightweight setup in an easy way to load test your web apps.

Not Every Goroutine Belongs in a Pool

A pool is for a batch of similar work. Some situations want something else:

Waiting on several different things at once — no limit needed, just a group:

g, ctx := errgroup.WithContext(ctx)

var user *User
var orders []Order

g.Go(func() (err error) { user, err = loadUser(ctx, id); return })
g.Go(func() (err error) { orders, err = loadOrders(ctx, id); return })

if err := g.Wait(); err != nil {
    return nil, fmt.Errorf("load profile: %w", err)
}

Three sequential 100ms calls become one 100ms call. This is the highest-value use of errgroup in a typical request handler, and it needs no pool at all.

Work that must happen in order — a pool is the wrong shape entirely; you want a single consumer, like the simple queue implementation I wrote about earlier.

Fire-and-forget background work — resist it. A goroutine started in a request handler outlives the request, holds whatever it captured, and will be killed mid-flight when the process shuts down. If it matters, it belongs in a durable queue; if it does not, do it inline. The same reasoning applies at shutdown time, which I covered in graceful shutdown in Go web services.

Only trying if there is capacity — TryGo starts the goroutine only if a slot is free, and reports whether it did:

if !g.TryGo(func() error { return prefetch(ctx, url) }) {
    // Pool is busy; skip this optional work rather than blocking.
    metrics.PrefetchSkipped.Inc()
}

Pitfalls

Panic recovery is worth spelling out, because one bad input taking down the whole process is a common way for batch jobs to fail:

g.Go(func() (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("panic processing %s: %v", url, r)
        }
    }()
    return process(ctx, url)
})

The named return value err is what makes this work — the deferred function assigns to it after the panic is recovered.

Conclusion

The pattern is small: pick a limit that matches your real bottleneck, use errgroup.WithContext so failures cancel their siblings, return errors instead of logging them, and give every blocking operation a context. Most of the time that is eight lines and no channel plumbing at all. Save the hand-rolled channel pool for the cases where you genuinely need to stream results or vary the shape of the work — and when you do write one, remember to close the jobs channel.

One place this pattern turns up more than you would expect: running the tool calls an LLM asks for, several at a time but not unboundedly — see tool use in Go. And when a pool does leak a worker, Go 1.27’s goroutine leak profile will now name it — what’s new in Go 1.27.

Early hints is a special HTTP header that is sent before the web server sends the final HTTP response to the client. At this moment it’s supported only by Chrome browser. As soon as the browser requests a page, server immediately returns 103 early hints header. In the meantime, a server will generate a usual HTTP response. This helps us utilize in maximum the loading time by letting browser know what resources it should preload while waiting for the final response from a server.

Enough theory, let’s write some code :)

First, I’m going to create an index.html with some dummy structure. Also, I will load bootsrap frontend framework to simulate some heavy css and js references during page load.

<html>
  <head>
    <title>Hello!</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-Zenh87qX5JnK2Jl0vWa8Ck2rdkQ2Bzep5IDxbcnCeuOxjzrPF/et3URy9Bv1WTRi" crossorigin="anonymous">
  </head>
  <body>
  <div class="p-2 bg-success">
    <h1 class="text-white">Hello!</h1>
  </div>
  <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-OERcA2EqjJCMA+/3y+gxIOqMEjwtxJY7qPCqsdltbNJuaOe923+mo//f6V8Qbsw3" crossorigin="anonymous"></script>
  </body>
</html>

Now we need to serve it. Let’s create a server.

//go:embed index.html
var index string // embeded index.html

func main() {
	log.Println("Starting server...")
    // Handler for the page without early hints.
	http.HandleFunc("/1", noHintsHandler)
    // Handler for the page with early hints
	http.HandleFunc("/2", withHintsHandler)
	if err := http.ListenAndServe(":8082", nil); err != nil {
		log.Fatal(err)
	}
}

func noHintsHandler(w http.ResponseWriter, r *http.Request) {
	t := template.New("")
	t, err := t.Parse(index)
	if err != nil {
		log.Println(err)
	}
	t.Execute(w, nil)
}

func withHintsHandler(w http.ResponseWriter, r *http.Request) {
    // Adding headers with preload information for bootstrap.min.css and bootstrap.bundle.min.js
	w.Header().Add("Link", "<https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/css/bootstrap.min.css>; rel=preload; as=style")
	w.Header().Add("Link", "<https://cdn.jsdelivr.net/npm/bootstrap@5.2.2/dist/js/bootstrap.bundle.min.js>; rel=preload; as=script")
    // Sending 103 status code.
	w.WriteHeader(http.StatusEarlyHints)

    t := template.New("")
	t, err := t.Parse(index)
	if err != nil {
		log.Println(err)
	}
    // Sending 200 status code.
	w.WriteHeader(http.StatusOK)
	t.Execute(w, nil)
}

Now it’s time to see our pages in action! Run our server go run main.go, open the page without early hints http://localhost:8082/1 in Chrome, open inspector, go to Lighthouse tab and click on “Analyze page load” button. And this is what we can see: It takes a while until bootstrap resources got loaded by a browser. As result, FCP (First Contentful Paint) is 1492,8ms.

Now, let’s do the same for the page with the early hints http://localhost:8082/2 And this is a result: As you can see, the page loaded much faster now. Bootstrap dependencies (bootstrap.min.css and bootstrap.bundle.min.js) were preloaded in the beginning and FCP now is 437,8ms. More than 3 times faster, quite an impressive result!

However, it does not mean that you have to preload absolutely all resources now. Just try to experiment with these things, see how it affects your page performance and decide for yourself the right balance.

You can find the source code here.

If you want to measure the difference on your own service rather than take my numbers for it, an easy way to load test your web apps shows the setup I use. And for the wins that happen before the request even reaches your handler, have a look at how to make Nginx cache cookie aware.

Let’s write a simple program, which generates a map of numbers and print them to the console:

package main

import "log"

var numbers = make(map[int]int, 100)

func main() {
	generateNumbersMap()
}

func generateNumbersMap() {
    // Write.
	for i := 0; i < 100; i++ {
		numbers[i] = i
	}
    // Read.
	for i := 0; i < 100; i++ {
		log.Print(numbers[i])
	}
}

Now, if we run it with the data race detector option go run -race main.go, we can see the printed list of numbers in the console without any data race problems.

Everything seems to be good. Is it? Let’s add some concurrency to our super complex program 😄 and see what happens:

package main

import (
	"log"
	"sync"
)

var numbers = make(map[int]int, 100)

func main() {
	generateNumbersMap()
}

func generateNumbersMap() {
	wg := sync.WaitGroup{}
    // Write.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			numbers[i] = i
		}(i)
	}
    // Read.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			log.Print(numbers[i])
		}(i)
	}

	wg.Wait()
}

If we run it now, in the console we can notice the data race errors:

==================
WARNING: DATA RACE
Write at 0x00c0001241b0 by goroutine 8:
  runtime.mapassign_fast64()
      /usr/local/opt/go/libexec/src/runtime/map_fast64.go:92 +0x0
  main.generateNumbersMap.func1()
      /dev/webdevstation/blog-examples/concurent-map/main.go:68 +0xa4

Previous write at 0x00c0001241b0 by goroutine 7:
  runtime.mapassign_fast64()
      /usr/local/opt/go/libexec/src/runtime/map_fast64.go:92 +0x0
  main.generateNumbersMap.func1()
      /dev/webdevstation/blog-examples/concurent-map/main.go:68 +0xa4

Goroutine 8 (running) created at:
  main.generateNumbersMap()
      /dev/webdevstation/blog-examples/concurent-map/main.go:66 +0xb5
  main.main()
      /dev/webdevstation/blog-examples/concurent-map/main.go:33 +0x2f

Goroutine 7 (finished) created at:
  main.generateNumbersMap()
      /dev/webdevstation/blog-examples/concurent-map/main.go:66 +0xb5
  main.main()
      /dev/webdevstation/blog-examples/concurent-map/main.go:33 +0x2f
==================
==================
WARNING: DATA RACE
Read at 0x00c000146438 by goroutine 41:
  main.generateNumbersMap.func2()
      /dev/webdevstation/blog-examples/concurent-map/main.go:75 +0xc7

Previous write at 0x00c000146438 by goroutine 7:
  main.generateNumbersMap.func1()
      /dev/webdevstation/blog-examples/concurent-map/main.go:68 +0xb9

Goroutine 41 (running) created at:
  main.generateNumbersMap()
      /dev/webdevstation/blog-examples/concurent-map/main.go:73 +0x110
  main.main()
      /dev/webdevstation/blog-examples/concurent-map/main.go:33 +0x2f
==================

Found 2 data race(s)
exit status 66

There are several strategies that could be used to solve it. I will show one of them. We are going to introduce a new struct that provides it’s own mutex:

type SafeNumbers struct {
	sync.RWMutex
	numbers map[int]int
}

To be able to read and write items concurrently to this structure, we need to create the responsible methods:

func (sn *SafeNumbers) Add(num int) {
	sn.Lock()
	defer sn.Unlock()
	sn.numbers[num] = num
}

Here we are basically telling to lock the numbers map, during adding of the new number to it. Other goroutines will wait until it became unlocked again.

And another method for reading:

func (sn *SafeNumbers) Get(num int) (int, error) {
	sn.RLock()
	defer sn.RUnlock()
	if number, ok := sn.numbers[num]; ok {
		return number, nil
	}
	return 0, errors.New("Number does not exists")
}

Next, let’s refactor our generateNumbersMap() function:

func generateNumbersMap() {
	wg := sync.WaitGroup{}
    // Init our "safe" numbers map struct.
	safeNumbers := &SafeNumbers{
		numbers: map[int]int{},
	}
    // Write.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			safeNumbers.Add(i)
		}(i)
	}
    // Read.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			number, err := safeNumbers.Get(i)
			if err != nil {
				log.Print(err)
			} else {
				log.Print(number)
			}
		}(i)
	}

	wg.Wait()
}

If we run go run -race main.go now, there will be no more data race issues!

As I mentioned before, there also other ways to solve it. One of them is using of a special go type sync.Map.

Nevertheless, I hope this was helpful and you know now how to work safely with the maps in go. Especially, you should be careful with them when you create the web services, because every http request initiating a new goroutine.

As usual, the source code you can find here.

If the goroutines writing to that map came from a batch of work, the next thing to fix is usually how many of them there are at once — worker pools in Go with errgroup covers bounding that. And to cancel them cleanly when the request goes away, see understanding Golang context.

Ristretto is a fast, concurrent cache library built with a focus on performance and correctness.

This library was created by the Dgraph team as a contention-free cache for the Dgraph database.

Let’s dive into the practical example. We are going to build a simple application that gets a list of users from the database. In the first iteration, there will be no caching layer at all. In the second iteration, we will add a Ristretto caching and compare execution time.

Below, you can see that I defined a repository package with the Repository interface and with InMemoryRepository implementation:

package repository

import "fmt"

// Repository interface to handle users data.
type Repository interface {
	GetUsers() map[int]string
}

type InMemoryRepository struct{}

// NewInMemoryRepository constructs and returns InMemoryRepository.
func NewInMemoryRepository() *InMemoryRepository {
	return &InMemoryRepository{}
}

// GetUsers returns 50000 dummy users from the in-memory repository.
func (r *InMemoryRepository) GetUsers() map[int]string {
	users := make(map[int]string)
	for i:=1; i <= 50000; i++ {
		users[i] = fmt.Sprintf("User %d", i)
	}

	return users
}

Next, we are going to call a GetUsers() method 100 times to simulate calling of the same function from several places in the real-world applications:

package main

import (
	"fmt"
	"github.com/alexsergivan/blog-examples/ristretto/repository"
)

func main() {
	for i:=0; i<100; i ++ {
		UsersGetter(repository.NewInMemoryRepository())
	}
}

func UsersGetter(repository repository.Repository) {
	repository.GetUsers()
}

Let’s measure how much time it takes to execute it with time go run main.go:

1.46s user
0.34s system
106% cpu
1.686 total

Next, we are going to add a caching layer to our application.

Don’t forget to get the Ristretto library:

  go get github.com/dgraph-io/ristretto

Inside repository package we inject Ristretto cache:

package repository

import (
	"fmt"
	"github.com/dgraph-io/ristretto"
	"time"
)

type InMemoryRepository struct{
	cache        *ristretto.Cache
}

// NewInMemoryRepository constructs and returns InMemoryRepository.
func NewInMemoryRepository(ristrettoCache *ristretto.Cache) *InMemoryRepository {
	return &InMemoryRepository{
		cache: ristrettoCache,
	}
}

// GetUsers returns 50000 dummy users from the in-memory repository.
func (r *InMemoryRepository) GetUsers() map[int]string {
	key := "users"
	value, found := r.cache.Get(key)
    // If the users data not cached yet, get it from the repository.
	if !found {
		users := make(map[int]string)
		for i:=1; i <= 50000; i++ {
			users[i] = fmt.Sprintf("User %d", i)
		}
        // Adds data to the cache for 1h.
		r.cache.SetWithTTL(key, users, 1, 1*time.Hour)
		time.Sleep(10 * time.Millisecond)
		return users
	}
	return value.(map[int]string)
}

Next, inside the main() function we initiate a new Ristretto cache and pass it to the InMemoryRepository:

  package main

import (
	"github.com/alexsergivan/blog-examples/ristretto/repository"
	"github.com/dgraph-io/ristretto"
)

func main() {
	ristrettoCache, _ := ristretto.NewCache(&ristretto.Config{
		NumCounters: 1e7,     // Num keys to track frequency of (10M).
		MaxCost:     1 << 30, // Maximum cost of cache (1GB).
		BufferItems: 64,      // Number of keys per Get buffer.
	})

	for i:=0; i<100; i ++ {
		UsersGetter(repository.NewInMemoryRepository(ristrettoCache))
	}
}

func UsersGetter(repository repository.Repository) {
	repository.GetUsers()
}

Let’s check how much time it takes to perform the same action:

0.29s user
0.26s system
147% cpu
0.377 total

As you can notice, the total time is 4 times less than in the example without caching layer.

Despite a silly example, I hope you got an idea of how to integrate the Ristretto caching into your application and how it could improve overall performance.

The complete source code you can find here.

Ristretto is safe for concurrent use, which is exactly the problem a plain map does not solve — see concurrent map writing and reading in Go if you want the failure mode in detail. I also refactored this caching layer with generics in how Golang generics minimize the amount of code you need to write. For a cache with a very different failure mode — one that costs you money rather than latency when it silently stops working — see prompt caching.