How can I concurrently download files, limited to 3 at the same time

Question

I am currently downloading files sequentially like this:

for _, url := range urlSet.URLs {
  bytes, err := getXML(url.Loc)
}

How can I do this concurrently in golang, but limit it to 3 at a time.

Answer 1

The following code will create 3 workers inside goroutines, each one continually grabbing URLs from the jobs channel, and writing results to the results channel.

In this example, the results are meaningless and ignored, but you might want to keep track of errors for retry later, or maybe something else all together.

If you don't care about results at all, then feel free to remove the results channel from this code, which would make it slightly more efficient.

package main

import (
    "fmt"
    "time"
)

func worker(id int, jobs <-chan URL, results chan<- int) {
    for j := range jobs {
        bytes, err := getXML(j.Loc)
        results <- 0 // flag that job is finished
    }
}

func main() {
    //
      // Get URLSET HERE
    //
    numJobs := len(urlSet.URLs)
    jobs := make(chan URL, numJobs)
    results := make(chan int, numJobs)

    for w := 1; w <= 3; w++ { // only 3 workers, all blocked initially
        go worker(w, jobs, results)
    }

    // continually feed in urls to workers
    for _, url := range urlSet.URLs {
        jobs <- url
    }
    close(jobs) // no more urls, so tell workers to stop their loop

    // needed if you want to make sure that workers don't block forever on writing results, remove both this loop and workers writing results if you don't need output from workers
    for a := 1; a <= numJobs; a++ {
        <-results
    }
}