The Token Bucket, in One Paragraph

Picture a bucket that holds b tokens and refills at r tokens per second. Every request takes one token. If the bucket is empty, the request is rejected (or waits). That is the whole model, and it has one property that makes it the right default: b is a burst allowance. A client that has been idle can spend its accumulated tokens all at once, then settles into the steady rate. Real traffic is bursty — a page load firing six API calls should not be punished, while a loop firing six hundred should.

import "golang.org/x/time/rate"

// 10 requests per second, bursts of up to 20.
limiter := rate.NewLimiter(10, 20)

if !limiter.Allow() {
    // over budget
}

Three methods, for three different situations:

rate.Limit is a float, so fractional rates work: rate.Every(time.Minute/100) is 100 per minute, and rate.Limit(0.5) is one request every two seconds. rate.Inf disables limiting entirely, which is handy for a per-plan configuration where some tier is unlimited.

A Global Limiter Is Not Enough

The naive version puts one limiter in front of everything:

var global = rate.NewLimiter(100, 200)

func limit(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !global.Allow() {
            http.Error(w, "too many requests", http.StatusTooManyRequests)
            return
        }
        next.ServeHTTP(w, r)
    })
}

This protects your server but not your users: one aggressive client can still eat the entire global budget and everyone else gets 429s. A global limiter is a useful backstop, not a fairness mechanism. What you want is a limiter per client, with the global one behind it.

Per-Client Limiters

Keep a map from client key to limiter, guarded by a mutex, with a janitor that evicts idle entries so the map does not grow forever.

package ratelimit

import (
    "sync"
    "time"

    "golang.org/x/time/rate"
)

type client struct {
    limiter  *rate.Limiter
    lastSeen time.Time
}

// Store hands out one limiter per key and forgets keys that go quiet.
type Store struct {
    mu      sync.Mutex
    clients map[string]*client
    rate    rate.Limit
    burst   int
    ttl     time.Duration
}

func NewStore(r rate.Limit, burst int, ttl time.Duration) *Store {
    s := &Store{
        clients: make(map[string]*client),
        rate:    r,
        burst:   burst,
        ttl:     ttl,
    }
    go s.cleanup()
    return s
}

// Limiter returns the limiter for key, creating it on first use.
func (s *Store) Limiter(key string) *rate.Limiter {
    s.mu.Lock()
    defer s.mu.Unlock()

    c, ok := s.clients[key]
    if !ok {
        c = &client{limiter: rate.NewLimiter(s.rate, s.burst)}
        s.clients[key] = c
    }
    c.lastSeen = time.Now()
    return c.limiter
}

func (s *Store) cleanup() {
    ticker := time.NewTicker(s.ttl)
    defer ticker.Stop()

    for range ticker.C {
        s.mu.Lock()
        for key, c := range s.clients {
            if time.Since(c.lastSeen) > s.ttl {
                delete(s.clients, key)
            }
        }
        s.mu.Unlock()
    }
}

The mutex is not optional. A bare map[string]*rate.Limiter written from concurrent handlers is a textbook data race, and Go’s runtime will happily crash the process with concurrent map writes — the same failure I dug into in concurrent map writing and reading in Go.

Two design notes. sync.Map is not a better fit here: it is optimised for read-mostly workloads with stable keys, and this map is written on every new client. And an unbounded map is a memory-exhaustion vector if the key is attacker-controlled — hence the TTL. For a hard cap, put an LRU in front of it.

The Middleware

func Middleware(store *Store) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            limiter := store.Limiter(clientKey(r))

            // Reserve, rather than Allow, so we can report Retry-After.
            res := limiter.Reserve()
            if !res.OK() {
                // Burst is smaller than the request size; never satisfiable.
                http.Error(w, "rate limit misconfigured", http.StatusInternalServerError)
                return
            }

            if delay := res.Delay(); delay > 0 {
                // We are not going to wait, so give the token back.
                res.Cancel()

                w.Header().Set("Retry-After", strconv.Itoa(int(math.Ceil(delay.Seconds()))))
                w.Header().Set("RateLimit-Limit", strconv.Itoa(limiter.Burst()))
                w.Header().Set("RateLimit-Remaining", "0")
                w.Header().Set("RateLimit-Reset", strconv.Itoa(int(math.Ceil(delay.Seconds()))))

                w.WriteHeader(http.StatusTooManyRequests)
                json.NewEncoder(w).Encode(map[string]string{
                    "error": "rate limit exceeded",
                })
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

res.Cancel() is the line people forget. Reserve takes the token immediately; if you then decide not to wait, cancelling returns it to the bucket. Skip it and every rejected request still consumes budget, so a client that trips the limit stays locked out far longer than intended.

This plugs into any router the same way as the handlers in my Go middleware example:

store := ratelimit.NewStore(rate.Limit(10), 20, 10*time.Minute)

r := chi.NewRouter()
r.Use(ratelimit.Middleware(store))

Choosing the Client Key

This is where rate limiting is usually got wrong, and it is worth more thought than the algorithm.

Authenticated requests: key on the identity. An API key or user ID is stable, meaningful, and cannot be spoofed once you have verified the token. If you are issuing JWTs — as in user authentication in Go Echo with JWT — the subject claim is your key.

Anonymous requests: key on the IP, carefully. r.RemoteAddr behind a proxy is the proxy’s address, so every user shares one bucket. But blindly trusting X-Forwarded-For is worse: it is a client-supplied header, and anyone can put whatever they like in it to get a fresh bucket per request.

func clientKey(r *http.Request) string {
    // Authenticated callers are keyed on identity.
    if userID, ok := auth.UserFrom(r.Context()); ok {
        return "user:" + userID
    }

    // Only trust the proxy header if the request came from our proxy,
    // and take the address the proxy appended — the rightmost hop.
    ip, _, _ := net.SplitHostPort(r.RemoteAddr)
    if isTrustedProxy(ip) {
        if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
            parts := strings.Split(xff, ",")
            ip = strings.TrimSpace(parts[len(parts)-1])
        }
    }
    return "ip:" + ip
}

The rightmost entry is the one your own proxy added; everything to its left came from the client and is unverifiable. If your platform provides a trusted header — Cloudflare’s CF-Connecting-IP, or the standard Forwarded from a proxy you control — prefer it.

One more refinement: not all endpoints are equal. POST /reports/export might cost a hundred times what GET /health does. AllowN and ReserveN let you charge by cost:

cost := 1
if r.Method == http.MethodPost && strings.HasPrefix(r.URL.Path, "/reports") {
    cost = 25
}
res := limiter.ReserveN(time.Now(), cost)

Just keep the burst at least as large as your most expensive operation, or res.OK() returns false forever and that endpoint becomes permanently unreachable.

Limiting Yourself, Too

Rate limiting is not only defensive. When you are the client of somebody else’s API, respecting their limit proactively beats absorbing 429s and retrying. Wait is built for this:

type Client struct {
    http    *http.Client
    limiter *rate.Limiter
}

func NewClient() *Client {
    return &Client{
        http:    &http.Client{Timeout: 10 * time.Second},
        // The upstream allows 5 requests/second; stay under it.
        limiter: rate.NewLimiter(5, 5),
    }
}

func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    // Blocks until a token is available, or ctx is cancelled.
    if err := c.limiter.Wait(ctx); err != nil {
        return nil, fmt.Errorf("rate limiter: %w", err)
    }
    return c.http.Do(req.WithContext(ctx))
}

Wait returns an error if the context is cancelled or if its deadline arrives before a token would — so a caller that has already given up never sits in the queue. That is the context machinery doing exactly what it is for.

This composes neatly with a bounded worker pool: the pool caps how many requests are in flight, the limiter caps how many start per second. They constrain different things and you usually want both, as I covered in worker pools in Go with errgroup.

Where This Approach Stops Working

Be honest about the limits of an in-process limiter.

It is per instance. Three replicas with a limit of 10/s allow 30/s in total, and a client bouncing between them gets a fresh bucket each time. For a real global limit you need shared state — Redis with a Lua script that does the token accounting atomically, or a limiter at the edge.

It is lost on restart. Every deploy resets every bucket. Usually fine; occasionally not.

It costs you a request. The request still reaches your process, gets routed, and allocates before being rejected. Under a genuine flood, that is exactly the work you cannot afford — which is why volumetric protection belongs at the CDN or load balancer, not in your handler.

My rule of thumb: application limits enforce fairness and per-plan quotas; edge limits absorb abuse. They solve different problems and you want both. The nginx layer is a natural place for the coarse one, and it can be surprisingly nuanced — the cookie-aware caching tricks work the same way for keying limit_req zones.

limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;

location /api/ {
    limit_req zone=api burst=200 nodelay;
    proxy_pass http://backend;
}

Testing It

Rate limiting is easy to test badly, because time.Now() is involved. Keep the rates small and explicit rather than sleeping through real seconds:

func TestLimiterRejectsBurst(t *testing.T) {
    store := ratelimit.NewStore(rate.Limit(1), 3, time.Minute)
    h := ratelimit.Middleware(store)(http.HandlerFunc(
        func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) },
    ))

    codes := make([]int, 5)
    for i := range codes {
        req := httptest.NewRequest(http.MethodGet, "/", nil)
        req.RemoteAddr = "203.0.113.7:1234"
        rec := httptest.NewRecorder()
        h.ServeHTTP(rec, req)
        codes[i] = rec.Code
    }

    // Burst of 3 succeeds, the rest are rejected.
    want := []int{200, 200, 200, 429, 429}
    if !slices.Equal(codes, want) {
        t.Errorf("got %v, want %v", codes, want)
    }
}

Then confirm the behaviour under real load before you trust the number. Pointing a load test at the endpoint and watching the ratio of 200s to 429s tells you whether your limit matches the traffic you actually get — the setup in an easy way to load test your web apps is enough for this.

Checklist

• Per-client limiters, not one global bucket, with a global one as backstop.
• Key on identity when authenticated; on a verified IP otherwise.
• Evict idle limiters so the map cannot grow without bound.
• res.Cancel() whenever you reject instead of waiting.
• Send Retry-After and RateLimit-* headers so good clients can behave.
• Burst at least as large as your most expensive weighted operation.
• Wait(ctx) on the client side of other people’s APIs.
• Volumetric protection at the edge, fairness in the application.

Conclusion

golang.org/x/time/rate is one of those packages that does exactly one thing and does it without ceremony. The algorithm is not the hard part — picking a sensible client key, giving tokens back when you reject, and being clear about what an in-process limiter can and cannot promise is where the real work is. Get those right and a single misbehaving script stops being everybody else’s problem. And if the API you are the client of happens to be a model provider, an agent loop is a remarkably efficient way to find its limits — see tool use in Go.