Rust threadpool crate how to execute on iterator?

Question

Cargo.toml

threadpool = "1.8.1"

I am using threadpool crate, just for learning purposes, if there are some better threadpool crates please inform me.

Code with problem, simplified version:

use threadpool::ThreadPool;

fn main() {
    let pool = ThreadPool::new(4);
    let v2 = vec!["a".to_string(), "b".to_string(), "c".to_string(), ];
     
    for i in v2.iter() { // .iter() is the problem
        pool.execute(move || println!("{}", i));
    }
    
    pool.join();
}

Error:

error[E0597]: `v2` does not live long enough
for i in v2.iter() { // .iter() is the problem
    |                  ^^^^^^^^^ borrowed value does not live long enough
pool.execute(move || println!("{}", i));
    |             --------------------------------------- argument requires that `v2` is borrowed for `'static`

It is working fine without .iter(). Reason why I use it, it is for demonstration purposes in my code I use iter and zip to iter thru 2 vec of same size both of type String eg. urls.iter().zip(files_t4.iter()).

I do understand why I have error, but have no ideas how to solve it ?
I know that this is similar to How can I pass a reference to a stack variable to a thread?, but except trying another crate, not sure is it possible to use std::thread::scope like in first example.
I am learning hot to download files in Rust, and now what to see dow it is done with threadpool.
Any ideas are appreciated.

@IvanC from what read on net regarding rayon, my understanding is that it is not intended to be used for IO bound task, more for CPU bound tasks. Is this correct ? — WebOrCode, Dec 14 '22 at 12:41
It's a thread pool, you can use it for IO-bound task if that's what you need, by sizing the threadpool differently than the default. It's not like your current use of threadpool is useful for IO, as I assume you're running on a machine with 4 cores so 4 workers is useful for CPU-bound workloads, not IO-bound workloads (where you want a number of workers much larger than the number of cores as the workers will generally be waiting for IO) — Masklinn, Dec 14 '22 at 12:47
@Masklinn I just started with 4 do understand that it is not optimum fo IO. Anyway is there some best practice for number of threads depending on number of file that you want to download ? I know that 100 is working fine, because I test it with max number of 100 files and each file have separate thread. — WebOrCode, Dec 14 '22 at 12:53

score 2 · Accepted Answer · answered Dec 14 '22 at 12:26

2

I do understand why I have error, but have no ideas how to solve it ?

Use .into_iter, or just iterate directly on the vector (does the same thing).

Because you're using iter, the iterator is borrowing, so i is of type &String, and thus triggers a lifetime error as the compiler does not know that the scope will outlive the threadpool.

not sure is it possible to use std::thread::scope like in first example.

It is, however you'll have to implement the "threadpool" by hand, as the threadpool crate does not support scoped threads / tasks. Alternatively, use Rayon which, aside from parallel iterators, provides explicit threadpool and scoped tasks. It also provides other interesting parallelism tools e.g. broadcast, join, and of course spawn.

answered Dec 14 '22 at 12:26

Masklinn

34,759
3
38
57

Yes, not it is fine, sometime I feel like worst fool on SO. Two questions. Fist because into_iter creates a consuming iterator, that is, one that moves each value out of the vector (from start to end). The vector cannot be used after calling this. What do you do in Rust if I want to use it again. Should I just make 2 copies or there is something better. Do not need it now, but would like to know. Second is it not that rayon is used fo CPU bound task not IO ? – WebOrCode Dec 14 '22 at 12:48
You need scoped threads/tasks, then – MeetTitan Dec 14 '22 at 14:05

Rust threadpool crate how to execute on iterator?

1 Answers1