Describe the worker pool pattern in Go.

5 minintermediategoworker-poolconcurrencypatterns

Quick Answer

A worker pool starts a fixed number of goroutines ("workers") that all read from a shared jobs channel and process work concurrently, bounding how much parallel work happens at once instead of spawning one goroutine per task unconditionally. This matters when tasks are numerous or resource-intensive (like making outbound HTTP calls), since unbounded goroutine spawning can overwhelm downstream systems or exhaust memory even though goroutines themselves are cheap. The pattern typically uses a jobs channel workers read from, a results channel workers write to, and a sync.WaitGroup (or closing the jobs channel) to signal when all work is done, giving you control over concurrency level via the number of worker goroutines started.

Detailed Answer

Spawning a goroutine per task is fine for a handful of tasks, but a worker pool caps concurrency when you have many tasks or a rate-sensitive downstream dependency.

A basic worker pool

func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
    defer wg.Done()
    for j := range jobs {           // exits automatically once jobs is closed and drained
        results <- j * j
    }
}

func main() {
    jobs := make(chan int, 100)
    results := make(chan int, 100)
    var wg sync.WaitGroup

    const numWorkers = 5
    for w := 1; w <= numWorkers; w++ {
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }

    for j := 1; j <= 20; j++ {
        jobs <- j
    }
    close(jobs)   // signals workers: no more jobs coming

    wg.Wait()
    close(results)

    for r := range results {
        fmt.Println(r)
    }
}

Why bound concurrency at all

If each of 10,000 tasks makes an outbound API call, spawning 10,000 goroutines at once could open 10,000 simultaneous connections, likely tripping rate limits or overwhelming the remote service. A worker pool with, say, 20 workers processes the same 10,000 tasks with at most 20 concurrent outbound calls, trading total throughput for predictable, bounded load.

Choosing the pool size

There's no universal right number. CPU-bound work often benefits from a pool size near runtime.NumCPU(). I/O-bound work (network calls, disk access) can often use a much larger pool, since workers spend most of their time waiting, not consuming CPU. The right size is usually found by benchmarking against the actual downstream constraint, like a rate limit or connection pool size.