How to have constant number of worker functions executing at all times?

Question

Lets say I have a list of 10 worker functions and I want 2 (or more) to be running at all times in parallel, advancing through the list when one finishes and then loop around and continue forever. So not to overload the server.

workers := make([]func(), 10)
for i := 0; i < 10; i++ {
  workers[i] = createWorker()
}

func createWorker() func() {
  return func() {
    fmt.Println("I am working")
    time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
  }
}

// My idea, keep sending workers to a buffered channel of size 2, 
// so when one finishes it's no longer filled up and another worker is sent
workerChan := make(chan func(), 2)
go func() {
  worker := <-workerChan
  worker()
}()

for {
  for _, worker := range workers {
    workerChan <- worker
  }
}

This runs the first worker function, and no more. Maybe the idea is correct and I need some guidance on how to implement it correctly.

I'm not sure what your code is trying to do here. Are you looking for an example like: https://gobyexample.com/worker-pools? — JimB, Jan 21 '21 at 20:57

score 1 · Answer 1 · answered Jan 21 '21 at 20:58

You were getting there.

func worker(ch chan func()) {
    // worker needs to read from a channel until channel is closed, 
    // then it will stop
    for work := range ch {
        work()
    }
}

func main() {
    workers := 2
    workerChan := make(chan func(), 10)

    for i := 0; i < workers; i++ {
        // start workers
        worker(workerChan)
    }
    
    // add work to channel
    workerChan <- func() {
        // do work
    }
}

If you wrap it in a struct, you can make a very general worker pool with it, executing whatever you give to it.

How to have constant number of worker functions executing at all times?

1 Answers1