Worker Pool in Go

Originally published on Substack.

Go processes goroutines without considering available memory, CPU, and other resources. When spawning a goroutine, Go allocates 2KB of memory that grows and shrinks as needed. While lightweight compared to other languages, this doesn’t mean spawning billions of goroutines simultaneously is feasible—servers have resource limits.

The worker pool is a concurrency pattern that allows us to limit the number of goroutines that run at the same time.

Image from itnext.io

Let’s see a basic example without any limits:

func main() {
  nums := []int{1, 2, 3, 4, 5}

  var wg sync.WaitGroup
  for num := range nums {
    wg.Add(1)
    go func(num int) {
      fmt.Println(num)
      wg.Done()
    }(num)
  }

  wg.Wait()
}

2 4 3 0 1

This spawns five goroutines simultaneously. To implement limits, you can manually partition work:

func main() {
  nums := []int{1, 2, 3, 4, 5}

  var wg sync.WaitGroup
  wg.Add(2)
  go printNums(nums[:3], &wg)
  go printNums(nums[3:], &wg)

  wg.Wait()
}

func printNums(nums []int, wg *sync.WaitGroup) {
  for _, num := range nums {
    fmt.Println(num)
  }

  wg.Done()
}

However, this approach becomes difficult to maintain when adjusting limits. A better solution uses channels.

Channels

Channel is Go’s data structure for communicating between goroutines.

func main() {
  message := make(chan string)

  go func() {
    message <- "ping"
  }()

  fmt.Println(<-message) // ping
}

The main function is itself a goroutine. In this example, an anonymous goroutine sends “ping” through the channel while the main goroutine receives it.

Attempting to receive without a sender causes deadlock:

func main() {
  message := make(chan string)

  fmt.Println(<-message) // fatal error: all goroutines are asleep - deadlock!
}

We can use for-range to read multiple values from a channel:

func main() {
  messages := make(chan string)

  go func() {
    messages <- "ping"
    messages <- "pong"
    close(messages)
  }()

  for msg := range messages {
    fmt.Println(msg)
  }
}

ping
pong

Closing the channel signals that no more messages will be sent. Alternatively, the sender can be in the main goroutine:

func main() {
  messages := make(chan string)

  go printer(messages)

  messages <- "ping"
  messages <- "pong"
  close(messages)
}

func printer(messages chan string) {
  for msg := range messages {
    fmt.Println(msg)
  }
}

ping
pong

Worker Pool Implementation

We will use the “job” term to identify a list of tasks. The “worker” is a term for identifying a goroutine.

func main() {
  nums := []int{1, 2, 3, 4, 5}
  jobs := make(chan int)
  pool := 2

  var wg sync.WaitGroup
  wg.Add(pool)

  // Spawn workers
  for i := 1; i <= pool; i++ {
    go func() {
      for job := range jobs {
        fmt.Println(job)
      }
      wg.Done()
    }()
  }

  // Generating jobs
  for _, num := range nums {
    jobs <- num
  }
  close(jobs)

  wg.Wait()
}

1 3 4 5 2

With two workers listening to the jobs channel, each job is processed by one worker. Let’s update the code to show which worker processes which job:

// Spawn workers
for i := 1; i <= pool; i++ {
  go func(worker int) {
    for job := range jobs {
      fmt.Printf("Job %d is done by worker %d\n", job, worker)
    }
    wg.Done()
  }(i)
}

Job 1 is done by worker 2
Job 2 is done by worker 1
Job 4 is done by worker 1
Job 5 is done by worker 1
Job 3 is done by worker 2

Go guarantees no duplications — two workers will never process the same job.

Adjusting the pool limit requires only changing one variable:

pool := 3

Job 1 is done by worker 3
Job 3 is done by worker 2
Job 5 is done by worker 2
Job 4 is done by worker 3
Job 2 is done by worker 1

---

That’s all for the article. We already see how to use the worker pool pattern in Go. I hope you can get something from reading this article. Happy hacking and see you in another one.