Graceful Shutdown in Go Web Services: Stop Dropping Requests on Deploy

The first time I deployed a Go service behind a rolling update, our error dashboard lit up on every single release. Nothing was broken — the new version was fine, the old version was fine. The problem was the half-second in between, where the old process died mid-request and a few dozen users got a connection reset. Fixing it took about twenty lines of code, and I have copied those twenty lines into every service since.

What Actually Happens on Shutdown

When your orchestrator wants a container gone, it sends SIGTERM and starts a countdown. If the process is still alive when the countdown ends, it gets SIGKILL, which cannot be caught.

A Go program with no signal handling takes the default action for SIGTERM: immediate termination. Every open connection is severed. Any request that was 90% done is simply gone — the client sees a reset, your retry budget takes the hit, and if that request was a payment you now have a support ticket.

Graceful shutdown means using the window between SIGTERM and SIGKILL to:

1. Stop accepting new connections.
2. Let in-flight requests finish.
3. Drain background workers.
4. Close databases, caches and queues.
5. Exit before the countdown runs out.

The Twenty Lines

package main

import (
    "context"
    "errors"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{
        Addr:              ":8080",
        Handler:           newRouter(),
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       15 * time.Second,
        WriteTimeout:      30 * time.Second,
        IdleTimeout:       60 * time.Second,
    }

    // ctx is cancelled the first time we receive SIGINT or SIGTERM.
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    go func() {
        slog.Info("listening", "addr", srv.Addr)
        // ListenAndServe always returns a non-nil error; ErrServerClosed
        // is the expected one after Shutdown.
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            slog.Error("listen failed", "err", err)
            os.Exit(1)
        }
    }()

    <-ctx.Done()
    stop() // restore default handling: a second Ctrl-C now kills us
    slog.Info("shutdown signal received, draining")

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        slog.Error("graceful shutdown failed, forcing close", "err", err)
        _ = srv.Close()
    }

    slog.Info("shutdown complete")
}

That is the whole pattern. A few details are load-bearing:

signal.NotifyContext instead of a channel. Since Go 1.16 this gives you a context.Context that cancels on the listed signals, which composes with everything else that already takes a context. If contexts and cancellation are new to you, understanding Golang context is the background reading.

Calling stop() after the first signal. It restores the default signal behaviour, so an impatient operator pressing Ctrl-C a second time gets an immediate exit instead of being ignored.

A fresh context for Shutdown. Deriving it from ctx would be a bug: ctx is already cancelled, so Shutdown would return instantly and drain nothing.

Checking for ErrServerClosed. ListenAndServe returns it on a clean shutdown. Treating that as a failure produces a scary log line on every normal deploy.

What Shutdown Does and Does Not Do

Server.Shutdown closes all open listeners, closes idle connections, and then waits for active ones to become idle. It returns when everything is drained or when its context expires — whichever comes first.

What it does not cover:

• Hijacked connections, including WebSockets. Shutdown does not wait for them, and it does not close them. You have to track and close them yourself.
• Background goroutines you started outside the request path. Nothing knows about them.
• Long-polling or streaming responses. These are “active” for as long as they stream, so they will hold the drain open until your timeout fires.

For WebSockets, the usual approach is to register a callback that broadcasts a close frame:

srv.RegisterOnShutdown(func() {
    hub.CloseAll(websocket.CloseServiceRestart, "server restarting")
})

RegisterOnShutdown callbacks run in their own goroutines as soon as Shutdown starts, so they get the whole drain window to do their work.

Draining Background Workers Too

Most real services do more than serve HTTP. If you have consumers, cron loops, or a queue like the one in my simple queue implementation, they need to finish too. errgroup keeps this readable:

import "golang.org/x/sync/errgroup"

func run(ctx context.Context) error {
    srv := &http.Server{Addr: ":8080", Handler: newRouter()}
    queue := NewQueue()

    g, gCtx := errgroup.WithContext(ctx)

    // 1. Serve HTTP.
    g.Go(func() error {
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            return fmt.Errorf("http server: %w", err)
        }
        return nil
    })

    // 2. Consume the queue until the group context is cancelled.
    g.Go(func() error {
        return queue.Consume(gCtx)
    })

    // 3. When anything cancels gCtx — a signal, or a failure in another
    //    goroutine — drain the HTTP server.
    g.Go(func() error {
        <-gCtx.Done()

        shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
        defer cancel()

        return srv.Shutdown(shutdownCtx)
    })

    return g.Wait()
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    if err := run(ctx); err != nil {
        slog.Error("service stopped", "err", err)
        os.Exit(1)
    }
    slog.Info("service stopped cleanly")
}

The nice property here is that failure propagates in both directions. A signal drains the HTTP server; a fatal error in the queue consumer also drains the HTTP server, because errgroup.WithContext cancels gCtx as soon as any goroutine returns an error.

The order of shutdown matters, and it is the reverse of startup: stop accepting work, finish what you have, then close the things that work depends on. Closing your database pool before draining HTTP guarantees a burst of errors from requests that were nearly done.

// After g.Wait() returns, nothing is still using these.
defer db.Close()
defer redis.Close()

The Load Balancer Problem

Here is the part that surprises people: even a perfectly graceful process can drop requests.

Between the moment your pod receives SIGTERM and the moment the load balancer stops sending it traffic, there is a gap. Endpoint updates propagate asynchronously — through the API server, to kube-proxy or an ingress controller, and finally to the actual routing table. During that gap the balancer is still sending new connections to a server that has already closed its listener. Those connections are refused.

The fix is to keep serving for a few seconds after the signal arrives:

<-ctx.Done()
slog.Info("signal received, waiting for load balancer to deregister")

// Keep serving while the endpoint removal propagates.
time.Sleep(5 * time.Second)

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)

It feels wrong to sleep on purpose, but it is the standard remedy, and Kubernetes has a hook for exactly this so you do not need it in your code:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 45

preStop runs before SIGTERM is sent, while the pod is already being removed from the endpoints list. By the time your process sees the signal, traffic has stopped arriving.

Whichever way you do it, keep the arithmetic straight:

preStop sleep (5s) + drain timeout (20s) + close time (2s) < terminationGracePeriodSeconds (45s)

If the total exceeds the grace period, you get SIGKILL mid-drain and you are back where you started. Give yourself real headroom — the default grace period is 30 seconds, which is not much once a slow request is in flight.

A readiness probe that starts failing on SIGTERM achieves the same thing more precisely:

var ready atomic.Bool

func init() { ready.Store(true) }

// /readyz
func readyz(w http.ResponseWriter, r *http.Request) {
    if !ready.Load() {
        http.Error(w, "shutting down", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

// On signal, before draining:
ready.Store(false)

Keep /healthz (liveness) returning 200 the whole time — if liveness fails during shutdown, the kubelet may kill the container instead of letting it drain.

Docker Gotchas

Two container-level mistakes will silently defeat everything above.

Shell-form CMD makes your process PID 2. Written as CMD ./server, Docker runs /bin/sh -c ./server. The shell is PID 1, receives SIGTERM, and does not forward it. Your server never hears a thing.

# Wrong — signals stop at the shell
CMD ./server

# Right — exec form, your binary is PID 1
CMD ["./server"]

docker stop waits 10 seconds by default. If your drain window is 20 seconds, you will be killed halfway through. Raise it: docker stop -t 45, or stop_grace_period: 45s in Compose.

Verifying It Works

Do not take it on faith — this is easy to test. Add a slow endpoint, start a request, and signal the process mid-flight:

mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
    select {
    case <-time.After(10 * time.Second):
        w.Write([]byte("finished\n"))
    case <-r.Context().Done():
        // The client gave up; Shutdown does not cancel request contexts.
        return
    }
})

./server &
curl -s localhost:8080/slow &     # starts a 10s request
sleep 1
kill -TERM %1                     # signal while it is in flight

A correct implementation prints finished after ten seconds and then exits. A broken one prints nothing and the curl reports a reset connection.

For the same check under real traffic, point a load test at the service and restart it mid-run — the technique from an easy way to load test your web apps works well here. With graceful shutdown in place your error count during a restart should be exactly zero; without it, you will see the exact number of requests that were in flight.

Note what Shutdown does not do: it does not cancel r.Context() for in-flight requests. That is deliberate — the request should be allowed to complete. It also means a handler with no timeout of its own can hold the drain open until your shutdown context expires, which is why the WriteTimeout in the first example matters.

Checklist

• signal.NotifyContext for SIGINT and SIGTERM.
• srv.Shutdown with its own fresh, bounded context.
• errors.Is(err, http.ErrServerClosed) treated as success.
• Background workers cancelled through the same context, drained before dependencies close.
• Dependencies closed last, in reverse order of startup.
• Readiness probe flipped to failing before the drain begins.
• preStop hook or a deliberate sleep to cover load balancer propagation.
• Grace period comfortably larger than the sum of your timeouts.
• Exec-form CMD in the Dockerfile.
• A test that proves an in-flight request survives a SIGTERM.

Conclusion

Graceful shutdown is one of those features nobody notices when it works — which is precisely the point. Twenty lines in main, a couple of timeouts that add up correctly, and a container that actually forwards signals will turn every deploy from a small burst of errors into a non-event. It is the cheapest reliability win available to a Go service, and worth adding before the next release rather than after the next incident.