Multiple threads Go for HTTP get

Question

I'm working with Go in an API for the bus frequency of my city, but i'm a little bit stuck on the threads when i try to make HTTP Get to many urls. Without concurrency, the programs takes over 16 minutes to complete the 1500 url calls to take the HTTP status code, and i was trying to use the concurrency, but after reading many posts i don't understand how goroutines work...

The idea is to make ONE function and change the number of requests, like here:

 go getBusPostStatus(600, 800)

But i'm completely stucked on that...

Here is the code:

package main

import (
"fmt"
"net/http"
"strconv"
"time"
)
var i int = 0
var convStr string
var message = make(chan string)

/*func main(){
    for i = 0; i < 1500; i++ {
        z = strconv.Itoa(i)
        url := "http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=" + z
        resp, err := http.Get(url)
        if err != nil {
            fmt.Println("Houston, we've got problems")
        }else{
            if resp.StatusCode == 200{
                fmt.Println("OK: El poste "+z+" existe")
            }else{
                fmt.Println("WARN: El poste "+z+" NO existe")
            }
        }
    }
}*/

//Return 2 houndred posts
func returnTH(c chan string){
 for i = 0; i < 200; i++ {  
    convStr = strconv.Itoa(i)
    url := "http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=" + convStr
    resp, err := http.Get(url)
    if err != nil {
        fmt.Println("Houston, we've got problems")
    }else{
        if resp.StatusCode == 200{
            //fmt.Println("OK: El poste "+z+" existe")
            c <- "OK: The bus post "+convStr+" exists"
        }else{
            //fmt.Println("WARN: El poste "+z+" NO existe")
            c <- "WARN: The bus post "+convStr+" does not exist"
        }   
    }
  }
}
func returnFH(z chan string){
  for i = 201; i < 400; i++ {   
   convStr = strconv.Itoa(i)
   url := "http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=" + convStr
    resp, err := http.Get(url)
    if err != nil {
        fmt.Println("Houston, we've got problems")
    }else{
        if resp.StatusCode == 200{
            //fmt.Println("OK: El poste "+z+" existe")
            z <- "OK: The bus post "+convStr+" exists"
        }else{
            //fmt.Println("WARN: El poste "+z+" NO existe")
            z <- "WARN: The bus post "+convStr+" does not exist"
        }   
    } 
  }
}

func threadPrint(c, z chan string){
   for {
       threadOne := <- c
       threadTwo := <- z
       fmt.Println(threadOne)
       fmt.Println(threadTwo)
   }
}
func main(){
    start := time.Now()
    var c chan string = make(chan string)
    var z chan string = make(chan string)
    //for i = 0; i < 1500; i++{
    go returnTH(c)
    go returnFH(z)
    go threadPrint(c,z)
    /*go getBusPostStatus(400, 600)
    go getBusPostStatus(600, 800)
    go getBusPostStatus(800, 1000)
    go getBusPostStatus(1000, 1200)
    go getBusPostStatus(1200, 1400)
    go getBusPostStatus(1400, 1500)*/
    //}
    timeExec:= time.Since(start)
    fmt.Println("Time to exec code = ", timeExec)

    /*var input string
    fmt.Scanln(&input)
    fmt.Println("done")*/
}

Many thanks in advance!!

It was an "idea" for make a thread every 200 calls to the URL, but I don't know how to make in threads, it was the "old system" without threads. I'm stucked on the way to make 1500 URL using threads, because the first function which is commented, the old /*main*/, takes over 16 minutes to take the 1500 responses... Thank you so much for helping!! — Brokes, Nov 18 '16 at 15:49
Go doesn't have threads, but if you mean goroutines, you're using them already by dispatching the function with the `go` keyword. — JimB, Nov 18 '16 at 15:51
Yeap, sorry, goroutines, but i don't know how to make many of them for doing the 1500 calls "in parallel" mode... Because without goroutines it takes 16 minutes to take the 1500 calls.... — Brokes, Nov 18 '16 at 15:56
1500 http calls in parallel is not going to be more efficient. see: https://stackoverflow.com/questions/37774624/go-http-get-concurrency-and-connection-reset-by-peer/37813844#37813844 — JimB, Nov 18 '16 at 16:03
Well, the idea is not to make the 1500 calls on a time, the idea is to make for example to this url:http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=2 the calls for checking if the post is alive or nope, from 0 to 1500, and i'm trying to make 3 or 4 calls on a time, if the bus post is alive, it will send HTTP->200, if is dead, HTTP-> 500, and i'm trying to get these status, but now i'm only doing 1 by 1 the calls to the URL, without threads... Thanks for the help! — Brokes, Nov 18 '16 at 16:37

score 4 · Accepted Answer · answered Nov 18 '16 at 16:05

Following is a simplified example code which requests 100 times concurrently and prints results, using goroutine and channel. Hope this code helps.

package main

import (
    "fmt"
    "math/rand"
    "time"
)

func main() {
    rep := 100
    results := make(chan string)

    // Use goroutine to send multiple time-consuming jobs to the channel.
    for i := 0; i < rep; i++ {
        go func(num int) {
            results <- mockHTTPRequest(num)
        }(i)
    }

    // Receive results from the channel and use them.
    for i := 0; i < rep; i++ {
        fmt.Println(<-results)
    }
}

func mockHTTPRequest(num int) string {
    timeDelay := rand.Intn(5000)
    time.Sleep(time.Duration(timeDelay) * time.Millisecond)
    if timeDelay%2 == 0 {
        return fmt.Sprintf("OK: The bus post %v exists", num)
    }
    return fmt.Sprintf("WARN: The bus post %v does not exist", num)
}

You can run this code on https://play.golang.org/p/RR34roRIl4 .

Thanks for helping! Well, the idea is: you make a call to an url, for example: http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=2, where the last number, in this case "2" is the number of the post, and in this case HTTP will return a HTTP -> 200 because this post exist... The problem is to make one by one, secuencially the 1500 calls from the 0 to the 1500... I don't know HOW to make the goroutines to make 3 or 4 calls at a time with Go "threads", from 0 to 1500... Many thanks for your help, bcoz i'm completely stucked since yesterday on this... — Brokes, Nov 18 '16 at 16:33

Multiple threads Go for HTTP get

1 Answers1