Process concurrent HTTP requests in Golang

Question

I'm trying to process a file which contains 200 URLs and use each URL to make an HTTP request. I need to process 10 URLs concurrently maximum each time (code should block until 10 URLs finish processing). Tried to solve it in go but I keep getting the whole file processed with 200 concurrent connection created.

for scanner.Scan() { // loop through each url in the file
        // send each url to golang HTTPrequest
        go HTTPrequest(scanner.Text(), channel, &wg)
}
fmt.Println(<-channel)
wg.Wait()

What should i do?

What would make it limit it to 10 goroutines? The number 10 is nowhere to be seen in your code. Check out this related question: [Is this an idiomatic worker thread pool in Go?](http://stackoverflow.com/questions/38170852/is-this-an-idiomatic-worker-thread-pool-in-go/38172204#38172204) — icza, Nov 21 '16 at 01:06

score 9 · Accepted Answer · answered Nov 21 '16 at 01:33

A pool of 10 go routines reading from a channel should fulfill your requirements.

work := make(chan string)

// get original 200 urls
var urlsToProcess []string = seedUrls() 

// startup pool of 10 go routines and read urls from work channel 
for i := 0; i<=10; i++ {
  go func(w chan string) {
     url := <-w
  }(work)
}

// write urls to the work channel, blocking until a worker goroutine
// is able to start work
for _, url := range urlsToProcess {
  work <- url
}

Cleanup and request results are left as an exercise for you. Go channels is will block until one of the worker routines is able to read.

Sorry to revisit the question, but what if you have bounded parallelism scenario, in which you need to read the URLs from a file and you do want to loop through the file line by line. — Jau L, May 13 '18 at 20:40

fengxsong · Answer 2 · 2016-11-23T02:51:28.333

2

code like this

longTimeAct := func(index int, w chan struct{}, wg *sync.WaitGroup) {
        defer wg.Done()
        time.Sleep(1 * time.Second)
        println(index)
        <-w
}
wg := new(sync.WaitGroup)
ws := make(chan struct{}, 10)
for i := 0; i < 100; i++ {
        ws <- struct{}{}
        wg.Add(1)
        go longTimeAct(i, ws, wg)
}
wg.Wait()

edited Nov 23 '16 at 02:51

answered Nov 22 '16 at 09:42

fengxsong

21
4

5

While this code may answer the question, providing additional context regarding why and/or how this code answers the question improves its long-term value. – Julien Lopez Nov 22 '16 at 10:11

Process concurrent HTTP requests in Golang

2 Answers2