What Actually Happens on Shutdown

When your orchestrator wants a container gone, it sends SIGTERM and starts a countdown. If the process is still alive when the countdown ends, it gets SIGKILL, which cannot be caught.

A Go program with no signal handling takes the default action for SIGTERM: immediate termination. Every open connection is severed. Any request that was 90% done is simply gone — the client sees a reset, your retry budget takes the hit, and if that request was a payment you now have a support ticket.

Graceful shutdown means using the window between SIGTERM and SIGKILL to:

1. Stop accepting new connections.
2. Let in-flight requests finish.
3. Drain background workers.
4. Close databases, caches and queues.
5. Exit before the countdown runs out.

The Twenty Lines

package main

import (
    "context"
    "errors"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{
        Addr:              ":8080",
        Handler:           newRouter(),
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       15 * time.Second,
        WriteTimeout:      30 * time.Second,
        IdleTimeout:       60 * time.Second,
    }

    // ctx is cancelled the first time we receive SIGINT or SIGTERM.
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    go func() {
        slog.Info("listening", "addr", srv.Addr)
        // ListenAndServe always returns a non-nil error; ErrServerClosed
        // is the expected one after Shutdown.
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            slog.Error("listen failed", "err", err)
            os.Exit(1)
        }
    }()

    <-ctx.Done()
    stop() // restore default handling: a second Ctrl-C now kills us
    slog.Info("shutdown signal received, draining")

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        slog.Error("graceful shutdown failed, forcing close", "err", err)
        _ = srv.Close()
    }

    slog.Info("shutdown complete")
}

That is the whole pattern. A few details are load-bearing:

signal.NotifyContext instead of a channel. Since Go 1.16 this gives you a context.Context that cancels on the listed signals, which composes with everything else that already takes a context. If contexts and cancellation are new to you, understanding Golang context is the background reading.

Calling stop() after the first signal. It restores the default signal behaviour, so an impatient operator pressing Ctrl-C a second time gets an immediate exit instead of being ignored.

A fresh context for Shutdown. Deriving it from ctx would be a bug: ctx is already cancelled, so Shutdown would return instantly and drain nothing.

Checking for ErrServerClosed. ListenAndServe returns it on a clean shutdown. Treating that as a failure produces a scary log line on every normal deploy.

What Shutdown Does and Does Not Do

Server.Shutdown closes all open listeners, closes idle connections, and then waits for active ones to become idle. It returns when everything is drained or when its context expires — whichever comes first.

What it does not cover:

• Hijacked connections, including WebSockets. Shutdown does not wait for them, and it does not close them. You have to track and close them yourself.
• Background goroutines you started outside the request path. Nothing knows about them.
• Long-polling or streaming responses. These are “active” for as long as they stream, so they will hold the drain open until your timeout fires.

For WebSockets, the usual approach is to register a callback that broadcasts a close frame:

srv.RegisterOnShutdown(func() {
    hub.CloseAll(websocket.CloseServiceRestart, "server restarting")
})

RegisterOnShutdown callbacks run in their own goroutines as soon as Shutdown starts, so they get the whole drain window to do their work.

Draining Background Workers Too

Most real services do more than serve HTTP. If you have consumers, cron loops, or a queue like the one in my simple queue implementation, they need to finish too. errgroup keeps this readable:

import "golang.org/x/sync/errgroup"

func run(ctx context.Context) error {
    srv := &http.Server{Addr: ":8080", Handler: newRouter()}
    queue := NewQueue()

    g, gCtx := errgroup.WithContext(ctx)

    // 1. Serve HTTP.
    g.Go(func() error {
        if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            return fmt.Errorf("http server: %w", err)
        }
        return nil
    })

    // 2. Consume the queue until the group context is cancelled.
    g.Go(func() error {
        return queue.Consume(gCtx)
    })

    // 3. When anything cancels gCtx — a signal, or a failure in another
    //    goroutine — drain the HTTP server.
    g.Go(func() error {
        <-gCtx.Done()

        shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
        defer cancel()

        return srv.Shutdown(shutdownCtx)
    })

    return g.Wait()
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    if err := run(ctx); err != nil {
        slog.Error("service stopped", "err", err)
        os.Exit(1)
    }
    slog.Info("service stopped cleanly")
}

The nice property here is that failure propagates in both directions. A signal drains the HTTP server; a fatal error in the queue consumer also drains the HTTP server, because errgroup.WithContext cancels gCtx as soon as any goroutine returns an error.

The order of shutdown matters, and it is the reverse of startup: stop accepting work, finish what you have, then close the things that work depends on. Closing your database pool before draining HTTP guarantees a burst of errors from requests that were nearly done.

// After g.Wait() returns, nothing is still using these.
defer db.Close()
defer redis.Close()

The Load Balancer Problem

Here is the part that surprises people: even a perfectly graceful process can drop requests.

Between the moment your pod receives SIGTERM and the moment the load balancer stops sending it traffic, there is a gap. Endpoint updates propagate asynchronously — through the API server, to kube-proxy or an ingress controller, and finally to the actual routing table. During that gap the balancer is still sending new connections to a server that has already closed its listener. Those connections are refused.

The fix is to keep serving for a few seconds after the signal arrives:

<-ctx.Done()
slog.Info("signal received, waiting for load balancer to deregister")

// Keep serving while the endpoint removal propagates.
time.Sleep(5 * time.Second)

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)

It feels wrong to sleep on purpose, but it is the standard remedy, and Kubernetes has a hook for exactly this so you do not need it in your code:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 45

preStop runs before SIGTERM is sent, while the pod is already being removed from the endpoints list. By the time your process sees the signal, traffic has stopped arriving.

Whichever way you do it, keep the arithmetic straight:

preStop sleep (5s) + drain timeout (20s) + close time (2s) < terminationGracePeriodSeconds (45s)

If the total exceeds the grace period, you get SIGKILL mid-drain and you are back where you started. Give yourself real headroom — the default grace period is 30 seconds, which is not much once a slow request is in flight.

A readiness probe that starts failing on SIGTERM achieves the same thing more precisely:

var ready atomic.Bool

func init() { ready.Store(true) }

// /readyz
func readyz(w http.ResponseWriter, r *http.Request) {
    if !ready.Load() {
        http.Error(w, "shutting down", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

// On signal, before draining:
ready.Store(false)

Keep /healthz (liveness) returning 200 the whole time — if liveness fails during shutdown, the kubelet may kill the container instead of letting it drain.

Docker Gotchas

Two container-level mistakes will silently defeat everything above.

Shell-form CMD makes your process PID 2. Written as CMD ./server, Docker runs /bin/sh -c ./server. The shell is PID 1, receives SIGTERM, and does not forward it. Your server never hears a thing.

# Wrong — signals stop at the shell
CMD ./server

# Right — exec form, your binary is PID 1
CMD ["./server"]

docker stop waits 10 seconds by default. If your drain window is 20 seconds, you will be killed halfway through. Raise it: docker stop -t 45, or stop_grace_period: 45s in Compose.

Verifying It Works

Do not take it on faith — this is easy to test. Add a slow endpoint, start a request, and signal the process mid-flight:

mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
    select {
    case <-time.After(10 * time.Second):
        w.Write([]byte("finished\n"))
    case <-r.Context().Done():
        // The client gave up; Shutdown does not cancel request contexts.
        return
    }
})

./server &
curl -s localhost:8080/slow &     # starts a 10s request
sleep 1
kill -TERM %1                     # signal while it is in flight

A correct implementation prints finished after ten seconds and then exits. A broken one prints nothing and the curl reports a reset connection.

For the same check under real traffic, point a load test at the service and restart it mid-run — the technique from an easy way to load test your web apps works well here. With graceful shutdown in place your error count during a restart should be exactly zero; without it, you will see the exact number of requests that were in flight.

Note what Shutdown does not do: it does not cancel r.Context() for in-flight requests. That is deliberate — the request should be allowed to complete. It also means a handler with no timeout of its own can hold the drain open until your shutdown context expires, which is why the WriteTimeout in the first example matters.

Checklist

• signal.NotifyContext for SIGINT and SIGTERM.
• srv.Shutdown with its own fresh, bounded context.
• errors.Is(err, http.ErrServerClosed) treated as success.
• Background workers cancelled through the same context, drained before dependencies close.
• Dependencies closed last, in reverse order of startup.
• Readiness probe flipped to failing before the drain begins.
• preStop hook or a deliberate sleep to cover load balancer propagation.
• Grace period comfortably larger than the sum of your timeouts.
• Exec-form CMD in the Dockerfile.
• A test that proves an in-flight request survives a SIGTERM.

Conclusion

Graceful shutdown is one of those features nobody notices when it works — which is precisely the point. Twenty lines in main, a couple of timeouts that add up correctly, and a container that actually forwards signals will turn every deploy from a small burst of errors into a non-event. It is the cheapest reliability win available to a Go service, and worth adding before the next release rather than after the next incident.

As an example, this is how I host whattoreadafter.xyz, a service that recommends books based on the book you just finished reading, by using AI.

Starting off, let’s enumerate the tools we’ll be using alongside Golang:

• DigitalOcean - a cloud computing platform that provides virtual machines and other resources.
• Docker - a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers.

In order to follow along, you will need to have a DigitalOcean account. If you don’t have one, you can sign up here and get $200 in credit over 60 days.

Let’s get started! For simplicity sake, our application will be a simple web server that returns current time in UTC format.

package main

import (
	"fmt"
	"net/http"
	"time"
)

func currentTimeHandler(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now().UTC()
	fmt.Fprintf(w, "Current Time (UTC): %s", currentTime.Format(time.RFC3339))
}

func main() {
	http.HandleFunc("/", currentTimeHandler)
	fmt.Println("Server is running on port 8080")
	http.ListenAndServe(":8080", nil)
}

Now let’s host it!

In order for you to easier follow along, I will list the steps we will take to host our application:

1. Prepare a Dockerfile for our application and place it in the root of our project alongside the main.go file.

Example:

FROM golang:alpine AS builder
RUN apk add --no-cache --update \
        git \
        ca-certificates
ADD . /app
WORKDIR /app
COPY go.mod ./
RUN  go mod download
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o /main .

FROM alpine
COPY --from=builder /main ./
RUN chmod +x ./main
ENTRYPOINT ["./main"]

2. Push your code to a GitHub repository. You can make it private if you want.

3. Go to DigitalOcean and create a new App. You can do so by clicking on the “Apps” tab in the left sidebar and then clicking “Create App”.

4. Select “GitHub” as your “Source” and click “Continue”.

5. Select the repository you want to deploy and click “Next”.

6. Select “Dockerfile” as your “Build type” and click “Next”.

7. Configure environment variables if you need to and click “Next”.

8. Select region for your app and click “Next”.

9. In the billing section, select the plan you want to use. You can start with the $5 per month plan and scale up later if you need to.

10. Click “Create Resources” and wait for your app to be deployed.

11. Once your app is deployed, you can access it by clicking on the “Live App” link.

That’s it! You have successfully deployed your Golang application!

You can also add your own domain name to your app by clicking on the “Settings” tab and then clicking on “Domains”.

Now, every time you push a new commit to your repository, DigitalOcean will automatically build and deploy your application. This is a great way to host your Golang applications, especially if you are just starting out and don’t want to spend a lot of money on hosting.

Thank you for reading! If you have any questions, feel free to reach out to me on Twitter.

Before you point real traffic at it, two things are worth having in place: graceful shutdown, so redeploys stop dropping requests, and a load test, so you know what the box can actually take.

When you develop web applications, it’s crucial to have a testing strategy. Nobody argues about the importance of unit, functional, and integration testing. Nevertheless, very often developers forget to test how their application works under high load. Even, when we have a “green light” from all our testing stages, including manual testing, better to not release it to production, until you load test it. Otherwise, nobody can guarantee, that application will work properly when 50 users will use it simultaneously.

In this article I’m going to load test our web application from the previous article.

For that, we are going to use a K6 tool, written in Go, and uses JavaScript for scripting.

k6 is a developer-centric, free and open-source load testing tool built for making performance testing a productive and enjoyable experience.

Let’s install this tool:

brew install k6

If you have different from the macOS operating system, please read about others ways to install here.

Next, we create a loadtests folder in the root of our project and inside we add a test.js file, where we are going to write our load tests scenarios:

import { sleep } from 'k6';
import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';

export let errorRate = new Rate('errors');

export let options = {
    // Here we define our scenarios.
    scenarios: {
        sign_in_page_test: {
            executor: 'constant-vus',
            duration: '1m',
            vus: 100, // amount of the virtual users
            tags: { test_type: 'signInPage' },
            exec: 'signInPage',
        },
    },
    // List of thresholds.
    thresholds: {
        http_req_duration: ['avg<500'], // avg response times must be below 0.5s
        errors: ['rate<0.1'], // <10% errors
    },
};

export function signInPage() {
    const res = http.get(getDomain() + '/user/signin');
    // Here we check the response status.
    const result = check(res, {
        'status is 200': (r) => r.status == 200,
    });
    // If it's different from 200, add info to the errorRate.
    errorRate.add(!result);
    sleep(3)
}

// Gets the domain from the environment variables.
function getDomain() {
    return __ENV.DOMAIN;
}

Above, we have declared a scenario to load test /user/signin page with 100 virtual users who continuously accessing our page in parallel.

To run this load test, we need to execute this command in the terminal:

k6 run --env DOMAIN="http://localhost:8777" ./loadtests/test.js

After some time we will see this results output:

As you could notice, all our defined thresholds were satisfied. So far so good!

Now, let’s see how we can integrate load testing to the Gitlab CI pipeline. Fortunately, it’s easy to do :)

Inside .gitlab-ci.yml we need to add this:

stages:
  - loadtest

loadtesting:
  stage: loadtest
  image:
    name: loadimpact/k6:latest
    entrypoint: [ '' ]
  variables:
    DOMAIN: '[your-testing-domain-here]'  
  script:
    - echo "executing K6 load tests in k6 container..."
    - k6 run --env DOMAIN=${DOMAIN} ./loadtests/test.js

That was it! I’ve described just an idea how you can easily integrate the load testing in your development routine. In the real-world situations you might create more complex load testing scenarios, which will help you find weak points of your application and prevent unexpected downtimes.

I wish you happy coding and no pagerduty calls during the night!😉

Once you can measure, you have something to optimise against. Two places I usually look first: 103 Early Hints in Go for front-end latency, and Ristretto caching for the expensive calls behind it. It is also the right tool to prove your graceful shutdown really does keep errors at zero during a deploy.

Let’s imagine a situation when you want to do an A/B test on your website, where 50% of users should see new headline text on the page. Other 50% of visitors will continue see the old page. In this case, all your server-side manipulation about splitting users to different versions will be ignored by nginx cache (of course, if you have it).

It happens because nginx, by default, has this configuration:

proxy_cache_key $scheme$proxy_host$request_uri;

So, it will use the same cache key for all users, who requests a page with the same url.

Luckily, nginx allows us easily to customize proxy_cache_key! What we need to do, it’s just to add a specific cookie to this key:

proxy_cache_key $scheme$proxy_host$request_uri$cookie_MY_COOKIE_NAME;

And that’s it! After this, if userA has MY_COOKIE_NAME=A and userB has MY_COOKIE_NAME=B they will receive different versions of page by the same url.

If you need more complex behaviour, where you won’t use hardcoded cookie name in your nginx config, you can do something like this:

if ($http_cookie ~* "ab_(.*?)=([\w-]+)" ) {
  set $abcookie $1$2;
}

proxy_cache_key $scheme$proxy_host$request_uri$abcookie; 

As you can see, you can use $http_cookie and generate proxy_cache_key based on specific cookie patterns. In this concrete example we check if http cookies contains cookie with regex pattern ab_(.*?)=([\w-]+) and if this cookie exists, we generate new variable for proxy_cache_key`.

Caching at the proxy is only one layer. For the one inside your Go process, see Ristretto — the most performant concurrent cache library for Go, and for shaving latency off the browser side, 103 Early Hints in Go.