Goroutines on WebDevStation

Worker Pools in Go: Bounded Concurrency with errgroup

Alex — Tue, 18 Aug 2026 11:20:00 +0200

Goroutines are so cheap that the first concurrent version of anything usually looks like for _, item := range items { go process(item) }. That works beautifully with ten items. With fifty thousand it opens fifty thousand database connections, and the thing you were trying to speed up falls over instead. What you almost always want is a bounded pool: N things in flight, no more. Here is how I build them.

The Problem With the Obvious Version

// Do not ship this.
func fetchAll(urls []string) {
    var wg sync.WaitGroup
    for _, url := range urls {
        wg.Add(1)
        go func() {
            defer wg.Done()
            fetch(url)
        }()
    }
    wg.Wait()
}

Three separate problems:

No limit. len(urls) concurrent requests. The remote service rate-limits you, or your file descriptors run out, or both.
No errors. fetch returns one and it goes nowhere.
No cancellation. If the caller gives up, every goroutine keeps running to completion.

The concurrency itself is not the mistake — the missing back pressure is.

The Classic Channel Pool

The traditional shape is a jobs channel, a fixed number of workers reading from it, and a results channel:

type job struct {
    ID  int
    URL string
}

type result struct {
    JobID int
    Body  []byte
    Err   error
}

func workerPool(ctx context.Context, jobs []job, workers int) []result {
    jobCh := make(chan job)
    resCh := make(chan result, len(jobs))

    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func(workerID int) {
            defer wg.Done()
            for j := range jobCh {
                body, err := fetch(ctx, j.URL)
                select {
                case resCh <- result{JobID: j.ID, Body: body, Err: err}:
                case <-ctx.Done():
                    return
                }
            }
        }(i)
    }

    // Feed the workers, stopping early if the caller cancels.
    go func() {
        defer close(jobCh)
        for _, j := range jobs {
            select {
            case jobCh <- j:
            case <-ctx.Done():
                return
            }
        }
    }()

    wg.Wait()
    close(resCh)

    out := make([]result, 0, len(jobs))
    for r := range resCh {
        out = append(out, r)
    }
    return out
}

This is worth understanding because you will read it in a lot of codebases, and because it shows the mechanics plainly. Note two things that are easy to get wrong:

close(jobCh) is the workers’ exit signal. for j := range jobCh ends when the channel closes. Forget the close and wg.Wait() blocks forever.
Every channel send is paired with <-ctx.Done(). Without that, a worker sending to a full resCh that nobody is reading leaks for the lifetime of the process.

It is also about forty lines to do something the standard extended library does in eight.

The errgroup Version

golang.org/x/sync/errgroup is a sync.WaitGroup that also collects the first error and cancels its siblings. SetLimit turns it into a bounded pool:

import "golang.org/x/sync/errgroup"

func fetchAll(ctx context.Context, urls []string) ([][]byte, error) {
    bodies := make([][]byte, len(urls))

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(10) // at most 10 in flight

    for i, url := range urls {
        g.Go(func() error {
            body, err := fetch(ctx, url)
            if err != nil {
                return fmt.Errorf("fetch %s: %w", url, err)
            }
            // Each goroutine owns exactly one slot: no mutex needed.
            bodies[i] = body
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }
    return bodies, nil
}

That is the whole pool. Behaviour worth knowing:

g.Go blocks once the limit is reached, until a slot frees up. The for loop becomes its own back pressure — no jobs channel needed.
errgroup.WithContext returns a derived context that is cancelled the moment any goroutine returns a non-nil error. Shadowing ctx with it, as above, is deliberate: every fetch gets the cancellable one.
g.Wait() returns the first error, and waits for the rest regardless. Later errors are discarded — if you need all of them, collect them yourself (errors.Join is a good fit, see error handling in Go).
Writing to bodies[i] is safe without a mutex because each goroutine writes one distinct element. Different elements of a slice are different memory; that is not a data race. Appending to a shared slice, or writing to a shared map, absolutely is — see concurrent map writing and reading in Go for what that failure looks like.

A Note on Loop Variables

The example above relies on Go 1.22’s per-iteration loop variables. Before 1.22, i and url were shared across iterations and every goroutine would see the final values — the single most common concurrency bug in Go. On older versions you must copy them:

for i, url := range urls {
    i, url := i, url // required before Go 1.22
    g.Go(func() error { /* ... */ })
}

Since Go 1.22 the copy is unnecessary. Leaving it in is harmless, and I still write it in code that must build on older toolchains. The loop semantics change was one of the more consequential recent additions to the language — I touched on the surrounding rules in mastering Golang for loops.

Streaming Results Instead of Preallocating

Indexing into a preallocated slice only works when you know the number of jobs up front. For a stream, send results down a channel and read them concurrently:

func processStream(ctx context.Context, in <-chan job, workers int) (<-chan result, func() error) {
    out := make(chan result)

    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(workers)

    go func() {
        // Closing `out` after every worker has finished lets the consumer
        // range over it and stop naturally.
        defer close(out)

        for j := range in {
            g.Go(func() error {
                body, err := fetch(ctx, j.URL)
                if err != nil {
                    return fmt.Errorf("job %d: %w", j.ID, err)
                }
                select {
                case out <- result{JobID: j.ID, Body: body}:
                    return nil
                case <-ctx.Done():
                    return ctx.Err()
                }
            })
        }
        _ = g.Wait()
    }()

    return out, g.Wait
}

The consumer ranges over out and then calls the returned function to get the error:

results, wait := processStream(ctx, jobs, 8)
for r := range results {
    save(r)
}
if err := wait(); err != nil {
    return fmt.Errorf("process stream: %w", err)
}

g.Wait() is safe to call more than once — subsequent calls return the same error immediately.

Picking the Limit

There is no universal number, but there is a reliable way to think about it.

CPU-bound work — parsing, hashing, image resizing, compression — saturates at roughly the number of cores. More goroutines just add scheduling overhead:

g.SetLimit(runtime.GOMAXPROCS(0))

I/O-bound work — HTTP calls, database queries, object storage — spends most of its time waiting, so the useful limit is much higher. But it is not “as high as possible”: it is whatever the slowest downstream dependency can absorb. If your database pool has 25 connections, a pool of 200 workers means 175 goroutines queueing on a mutex inside database/sql while your latency graph climbs.

// Match the constraint that actually binds.
g.SetLimit(db.Stats().MaxOpenConnections)

For outbound HTTP, remember that Go’s default transport keeps only 2 idle connections per host. Exceed that and you are opening a fresh TCP connection — plus a TLS handshake — for each extra request:

transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConnsPerHost = 50
transport.MaxConnsPerHost = 50
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

Then set the pool limit to match. Tuning one without the other gets you nothing.

Whatever you pick, measure it. Run the job at 5, 10, 25 and 50 and look at total wall time and downstream latency — the fastest setting for your batch is often the one that makes everything else on the system slower. Load testing is the honest way to find out; I wrote about a lightweight setup in an easy way to load test your web apps.

Not Every Goroutine Belongs in a Pool

A pool is for a batch of similar work. Some situations want something else:

Waiting on several different things at once — no limit needed, just a group:

g, ctx := errgroup.WithContext(ctx)

var user *User
var orders []Order

g.Go(func() (err error) { user, err = loadUser(ctx, id); return })
g.Go(func() (err error) { orders, err = loadOrders(ctx, id); return })

if err := g.Wait(); err != nil {
    return nil, fmt.Errorf("load profile: %w", err)
}

Three sequential 100ms calls become one 100ms call. This is the highest-value use of errgroup in a typical request handler, and it needs no pool at all.

Work that must happen in order — a pool is the wrong shape entirely; you want a single consumer, like the simple queue implementation I wrote about earlier.

Fire-and-forget background work — resist it. A goroutine started in a request handler outlives the request, holds whatever it captured, and will be killed mid-flight when the process shuts down. If it matters, it belongs in a durable queue; if it does not, do it inline. The same reasoning applies at shutdown time, which I covered in graceful shutdown in Go web services.

Only trying if there is capacity — TryGo starts the goroutine only if a slot is free, and reports whether it did:

if !g.TryGo(func() error { return prefetch(ctx, url) }) {
    // Pool is busy; skip this optional work rather than blocking.
    metrics.PrefetchSkipped.Inc()
}

Pitfalls

Pitfall	Fix
`g.Go` never returns	Something inside blocks forever — give every call a context and a timeout
Results come back in the wrong order	Index into a preallocated slice, or sort by an explicit sequence number
`panic` in a worker kills the process	Recover inside the goroutine and convert it to an error
Errors vanish	Return them from `g.Go`; do not just log them
`SetLimit` called after `g.Go`	Panics — set the limit before starting any work
Unbounded jobs channel eats memory	Use an unbuffered channel, or let `SetLimit` provide the back pressure
Shared map written from workers	`sync.Map`, a mutex, or per-worker maps merged at the end

Panic recovery is worth spelling out, because one bad input taking down the whole process is a common way for batch jobs to fail:

g.Go(func() (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("panic processing %s: %v", url, r)
        }
    }()
    return process(ctx, url)
})

The named return value err is what makes this work — the deferred function assigns to it after the panic is recovered.

Conclusion

The pattern is small: pick a limit that matches your real bottleneck, use errgroup.WithContext so failures cancel their siblings, return errors instead of logging them, and give every blocking operation a context. Most of the time that is eight lines and no channel plumbing at all. Save the hand-rolled channel pool for the cases where you genuinely need to stream results or vary the shape of the work — and when you do write one, remember to close the jobs channel.

One place this pattern turns up more than you would expect: running the tool calls an LLM asks for, several at a time but not unboundedly — see tool use in Go.

Concurrent Map Writing and Reading in Go, or how to deal with the data races.

Alex — Fri, 16 Jul 2021 11:10:38 +0200

This time, I will show you how to work with the maps in go effectively and prevent the occurrence of the data race errors. Data races happen when several goroutines access the same resource concurrently and at least one of the accesses is a write.

Let’s write a simple program, which generates a map of numbers and print them to the console:

package main

import "log"

var numbers = make(map[int]int, 100)

func main() {
	generateNumbersMap()
}

func generateNumbersMap() {
    // Write.
	for i := 0; i < 100; i++ {
		numbers[i] = i
	}
    // Read.
	for i := 0; i < 100; i++ {
		log.Print(numbers[i])
	}
}

Now, if we run it with the data race detector option go run -race main.go, we can see the printed list of numbers in the console without any data race problems.

Everything seems to be good. Is it? Let’s add some concurrency to our super complex program 😄 and see what happens:


package main

import (
	"log"
	"sync"
)

var numbers = make(map[int]int, 100)

func main() {
	generateNumbersMap()
}

func generateNumbersMap() {
	wg := sync.WaitGroup{}
    // Write.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			numbers[i] = i
		}(i)
	}
    // Read.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			log.Print(numbers[i])
		}(i)
	}

	wg.Wait()
}

If we run it now, in the console we can notice the data race errors:

==================
WARNING: DATA RACE
Write at 0x00c0001241b0 by goroutine 8:
  runtime.mapassign_fast64()
      /usr/local/opt/go/libexec/src/runtime/map_fast64.go:92 +0x0
  main.generateNumbersMap.func1()
      /dev/webdevstation/blog-examples/concurent-map/main.go:68 +0xa4

Previous write at 0x00c0001241b0 by goroutine 7:
  runtime.mapassign_fast64()
      /usr/local/opt/go/libexec/src/runtime/map_fast64.go:92 +0x0
  main.generateNumbersMap.func1()
      /dev/webdevstation/blog-examples/concurent-map/main.go:68 +0xa4

Goroutine 8 (running) created at:
  main.generateNumbersMap()
      /dev/webdevstation/blog-examples/concurent-map/main.go:66 +0xb5
  main.main()
      /dev/webdevstation/blog-examples/concurent-map/main.go:33 +0x2f

Goroutine 7 (finished) created at:
  main.generateNumbersMap()
      /dev/webdevstation/blog-examples/concurent-map/main.go:66 +0xb5
  main.main()
      /dev/webdevstation/blog-examples/concurent-map/main.go:33 +0x2f
==================
==================
WARNING: DATA RACE
Read at 0x00c000146438 by goroutine 41:
  main.generateNumbersMap.func2()
      /dev/webdevstation/blog-examples/concurent-map/main.go:75 +0xc7

Previous write at 0x00c000146438 by goroutine 7:
  main.generateNumbersMap.func1()
      /dev/webdevstation/blog-examples/concurent-map/main.go:68 +0xb9

Goroutine 41 (running) created at:
  main.generateNumbersMap()
      /dev/webdevstation/blog-examples/concurent-map/main.go:73 +0x110
  main.main()
      /dev/webdevstation/blog-examples/concurent-map/main.go:33 +0x2f
==================

Found 2 data race(s)
exit status 66

There are several strategies that could be used to solve it. I will show one of them. We are going to introduce a new struct that provides it’s own mutex:

type SafeNumbers struct {
	sync.RWMutex
	numbers map[int]int
}

To be able to read and write items concurrently to this structure, we need to create the responsible methods:

func (sn *SafeNumbers) Add(num int) {
	sn.Lock()
	defer sn.Unlock()
	sn.numbers[num] = num
}

Here we are basically telling to lock the numbers map, during adding of the new number to it. Other goroutines will wait until it became unlocked again.

And another method for reading:

func (sn *SafeNumbers) Get(num int) (int, error) {
	sn.RLock()
	defer sn.RUnlock()
	if number, ok := sn.numbers[num]; ok {
		return number, nil
	}
	return 0, errors.New("Number does not exists")
}

Next, let’s refactor our generateNumbersMap() function:

func generateNumbersMap() {
	wg := sync.WaitGroup{}
    // Init our "safe" numbers map struct.
	safeNumbers := &SafeNumbers{
		numbers: map[int]int{},
	}
    // Write.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			safeNumbers.Add(i)
		}(i)
	}
    // Read.
	for i := 0; i < 100; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			number, err := safeNumbers.Get(i)
			if err != nil {
				log.Print(err)
			} else {
				log.Print(number)
			}
		}(i)
	}

	wg.Wait()
}

If we run go run -race main.go now, there will be no more data race issues!

As I mentioned before, there also other ways to solve it. One of them is using of a special go type sync.Map.

Nevertheless, I hope this was helpful and you know now how to work safely with the maps in go. Especially, you should be careful with them when you create the web services, because every http request initiating a new goroutine.

As usual, the source code you can find here.

If the goroutines writing to that map came from a batch of work, the next thing to fix is usually how many of them there are at once — worker pools in Go with errgroup covers bounding that. And to cancel them cleanly when the request goes away, see understanding Golang context.

A Simple Queue Implementation in Golang with channels

Alex — Tue, 12 Jan 2021 18:40:24 +0000

In this post, I’m going to show the way how we can implement a simple queue in Golang, using channels.

Let’s clarify why would we ever use a queue? There many possible reasons. The most common one it’s when we have a list of operations (actions) and we need to process them one by one. This list either could be static or dynamic (when new items arriving continuously). As in the real-life, queue needed to be processed by something or someone. Imagine, a line of people in the supermarket nearby cashier.

Every customer in this line holds his own specific products and wants to pay for them. In other words, every queue item has it own set of data with instructions. A cashier serves his clients from a line one by one (processes queue items). What would happen if all customers decided to pay simultaneously? Probably, something like this:

To prevent this situation, let’s implement a queue in Go quickly :) First, we need to create a Queue struct and constructor function:

  // Queue holds name, list of jobs and context with cancel.
  type Queue struct {
     name   string
     jobs   chan Job
     ctx    context.Context
     cancel context.CancelFunc
  }

  // Job - holds logic to perform some operations during queue execution.
  type Job struct {
    Name   string
    Action func() error // A function that should be executed when the job is running.
  }

  // NewQueue instantiates new queue.
  func NewQueue(name string) *Queue {
    ctx, cancel := context.WithCancel(context.Background())

    return &Queue{
       jobs:   make(chan Job),
       name:   name,
       ctx:    ctx,
       cancel: cancel,
    }
  }

As you have seen above, I’ve declared an unbuffered channel (with no capacity) to hold Jobs in a Queue. Channels will give us a really powerful possibility to work in a concurrent environment. Moreover, our queue has a context with cancellation. This will help us to understand when all jobs in the queue were finished in all goroutines and after that, we can free all resources.

In the code below we are going to add some methods to the queue struct.

// AddJobs adds jobs to the queue and cancels channel.
func (q *Queue) AddJobs(jobs []Job) {
  var wg sync.WaitGroup
  wg.Add(len(jobs))

  for _, job := range jobs {
    // Goroutine which adds job to the queue.
    go func(job Job) {
      q.AddJob(job)
      wg.Done()
    }(job)
  }

  go func() {
    wg.Wait()
    // Cancel queue channel, when all goroutines were done.
    q.cancel()
  }()
}

// AddJob sends job to the channel.
func (q *Queue) AddJob(job Job) {
  q.jobs <- job
  log.Printf("New job %s added to %s queue", job.Name, q.name)
}

Next, let’s add a Run() method to the Job struct, which should execute a job’s action (the main purpose of the job).

// Run performs job execution.
func (j Job) Run() error {
  log.Printf("Job running: %s", j.GetName())

  err := j.Action()
  if err != nil {
    return err
  }

  return nil
}

Alright, now let’s create a worker, who will be responsible for serving our queue.

// Worker responsible for queue serving.
type Worker struct {
  Queue *Queue
}

// NewWorker initializes a new Worker.
func NewWorker(queue *Queue) *Worker {
  return &Worker{
    Queue: queue,
  }
}

// DoWork processes jobs from the queue (jobs channel).
func (w *Worker) DoWork() bool {
  for {
    select {
      // if context was canceled.
      case <-w.Queue.ctx.Done():
        log.Printf("Work done in queue %s: %s!", w.Queue.name, w.Queue.ctx.Err())
        return true
      // if job received.
      case job := <-w.Queue.jobs:
        err := job.Run()
        if err != nil {
          log.Print(err)
          continue
        }
    }
  }
}

Pay attention to the DoWork() method. There we have a for loop, which is listening for the jobs channel, and if the channel receives a job, it executes it, by running job.Run(). If the context was canceled, it means, that all jobs were executed and we can exit from the loop.

Now, let’s see how we can work with just created queue functionality on a real example. I’m going to create a simple program, where we need to import new products into our storage.

package main

import (
	"fmt"
	"github.com/alexsergivan/blog-examples/queue/queue"
)

// Our products storage.
var products = []string{
	"books",
	"computers",
}

func main()  {
    // New products, which we need to add to our products storage.
	newProducts := []string{
		"apples",
		"oranges",
		"wine",
		"bread",
		"orange juice",
	}
    // New queue initialization.
    productsQueue := queue.NewQueue("NewProducts")
	var jobs []queue.Job

    // Range over new products.
	for _, newProduct := range newProducts {
        // We need to do this, because variables declared in for loops are passed by reference.
        // Otherwise, our closure will always receive the last item from the newProducts.
        product := newProduct
        // Defining of the closure, where we add a new product to our simple storage (products slice)
		action := func() error {
			products = append(products, product)
			return nil
        }
        // Append job to jobs slice.
		jobs = append(jobs, queue.Job{
			Name:   fmt.Sprintf("Importing new product: %s", newProduct),
			Action: action,
		})
	}

    // Adds jobs to the queue.
	productsQueue.AddJobs(jobs)

    // Defines a queue worker, which will execute our queue.
    worker := queue.NewWorker(productsQueue)
    // Execute jobs in queue.
    worker.DoWork()

    // Prints products storage after queue execution.
	defer fmt.Print(products)
}

After executing this program, it will print out this data:

2021/01/12 22:10:33 New job Importing new product: apples added to NewProducts queue
2021/01/12 22:10:33 Job running: Importing new product: apples
2021/01/12 22:10:33 Job running: Importing new product: wine
2021/01/12 22:10:33 Job running: Importing new product: oranges
2021/01/12 22:10:33 Job running: Importing new product: bread
2021/01/12 22:10:33 Job running: Importing new product: orange juice
2021/01/12 22:10:33 New job Importing new product: orange juice added to NewProducts queue
2021/01/12 22:10:33 New job Importing new product: oranges added to NewProducts queue
2021/01/12 22:10:33 New job Importing new product: bread added to NewProducts queue
2021/01/12 22:10:33 New job Importing new product: wine added to NewProducts queue
2021/01/12 22:10:33 Work done in queue NewProducts: context canceled!
[books computers apples wine oranges bread orange juice]%

Please notice the last line in the logs: [books computers apples wine oranges bread orange juice] As you can see, the order is different from the order in newProducts slice. It happens because of goroutines nature. Sometimes one goroutine needs more time to finish its work than another. For the scenario, when the order is important, I will write a separate post later.

For now, that’s it :) Hope it was helpful!

The source code you can find here

A queue like this processes work in order, one item at a time. When order does not matter and you want several items in flight instead, reach for worker pools in Go with errgroup. Either way, make sure the consumer stops cleanly on shutdown — see graceful shutdown in Go web services.