Running thousands of goroutines concurrently

Question

I'm developing a simulator model for historical trading data. I have the below pseudo code where I'm using many goroutines.

start_day := "2015-01-01 09:15:00 "
end_day := "2015-02-01 09:15:00"
ch = make(chan float64)
var value bool

// pair_map have some int pairs like {1,2},{10,11} like approx 20k pairs which are taken
// dynamically from a range of input
for start_day.Before(end_day) {
    // a true value in ch is set at end of each func call
    go get_output(ch, start_day, pair_map)
    value <- ch
    start_day = start_date.Add(time.Hour * 24)
}

Now the problem here is that each get_otput() is executed approx. for 20k combinations per single day (which is taking approx. 1 min), here I'm trying to do it for a month time span using go routines (on single core machine), but it is taking almost same amount of time as running sequentially (approx. 30 min).

Is there anything wrong with my approach, or is it because I'm using a single core?

It looks to me as if you start a goroutine but then immediately wait until it is finished, therefore not actually running anything concurrently. — mkrieger1, Mar 29 '21 at 11:47
`value <- ch` is a blocking call - therefore you call `get_output` and wait for the result before the next iteration of your loop begins. If you want to run in parallel and wait afterwards it might be best to use a `WaitGroup` and then read all channels or write all results into a single channel (beware of the random order!) — Dominik, Mar 29 '21 at 11:51
If it's a single-core machine, there's no way for threading to improve performance, because only one can run at any given time. It's forced to be serialized by the available hardware. — Adrian, Mar 29 '21 at 13:25

score 3 · Accepted Answer · answered Mar 29 '21 at 12:23

I'm trying to do it for a month time span using go routines (on single core machine), but it is taking almost same amount of time as running sequentially

What makes you think it should perform any better? Running multiple goroutines has its overhead: goroutines have to be managed and scheduled, results must be communicated / gathered. If you have a single core, using multiple gorotuines cannot yield a performance improvement, only detriment.

Using goroutines may give a performance boost if you have multiple cores and the task you distribute are "large" enough so the goroutine management will be smaller than the gain of the parallel execution.

See related: Is golang good to use in multithreaded application?

Running thousands of goroutines concurrently

1 Answers1