How to execute a map of blocking http requests in parallel?

Question

I have a lot of code using ureq for http requests and I'm wondering if I can avoid using another http library.

I have a list of urls and I'm invoking ureq::get on them. I'm wondering if I can somehow make these calls in parallel. How would I create separate threads and execute these in parallel?

let v = vec!["www.qwe.com"; 3];
let responses = v.into_iter().map(|s| make_get_request(s)).collect::<Vec<_>>();

Masklinn · Accepted Answer · 2021-03-24T09:50:30.203

2

You can just use rayon. It's not ideal because it assumes CPU-bound work and will therefore spawn one thread per (logical) core by default, which may be less than you'd want for HTTP requests, but you could always customise the global threadpool (or run your work inside the scope of a local threadpool with a higher thread count).

edited Mar 24 '21 at 09:50

answered Mar 24 '21 at 09:01

Masklinn

34,759
3
38
57

Is customizing the number of threads a feature of rayon or is that done somehow else? – ditoslav Mar 24 '21 at 09:08
It's a feature of Rayon, check out the threadpoolbuilder. – Masklinn Mar 24 '21 at 09:13

score 0 · Answer 2 · answered Mar 24 '21 at 15:01

If you want as many threads as requests, you can just spawn them:

let v = vec!["www.qwe.com"; 3];
let handles = v
    .into_iter()
    .map(|s| thread::spawn(move || make_get_request(s)))
    .collect::<Vec<_>>();
let responses = handles.into_iter().map(|h| h.join()).collect::<Vec<_>>();

Playground

How to execute a map of blocking http requests in parallel?

2 Answers2