Concurrently apply the same mutable function to different elements of the same Vec

Question

I have a series of large structs I don't want to copy, and I want to apply a slow function to them . Since the slow function can be applied independently to each big struct, I want to do it concurrently.

All those big structs are in a container (a Vec is a good representation of that). I can't spawn a thread for each element of the Vec because borrowing one mutable element is the same as borrowing the whole vector each time.

struct SomeBigStruct {
    pub variable: i32,
}

impl SomeBigStruct {
    pub fn new() -> SomeBigStruct {
        SomeBigStruct { variable: 0 }
    }
}

fn some_slow_function(struct_in: &mut SomeBigStruct) {
    println!("{}", struct_in.variable);
    struct_in.variable += 1;
}

fn main() {
    let mut vec_of_big_struct: Vec<SomeBigStruct> = Vec::new();

    for i in 0..10 {
        vec_of_big_struct.push(SomeBigStruct::new());
    }
    for struct_to_edit in vec_of_big_struct.iter_mut() {
        some_slow_function(struct_to_edit);
    } //Compiles but is slow

    for struct_to_edit in vec_of_big_struct.iter_mut() {
        std::thread::spawn(move || {
            some_slow_function(struct_to_edit);
        });
    } //Can't compile because borrow checker isn't happy
}

Those aren't the actual struct or function, but the implementation details don't really matter.

What matters is that I don't want to copy the big struct because it is big, and I want to run the function concurrently on a mutable reference of that struct, knowing that each execution is independent.

Relevant questions: https://stackoverflow.com/q/52523237 https://stackoverflow.com/questions/33818141 — E_net4, Sep 29 '20 at 10:16

Masklinn · Accepted Answer · 2020-09-29T10:47:26.643

All those big structs are in a container (a Vec is a good representation of that). I can't spawn a thread for each element of the vec, because borrowing one mutable element is the same as borrowing the whole vector each time.

Not exactly, if you're indexing yes but that's because the borrow checker can't statically know that all your indices are different, so &mut v[a] and &mut v[b] could run with a == b and then broken because you'd have two mutable references to the same thing which is forbidden.

That's not an issue with iterators, you can borrow several items mutably simultanously: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=657d45a4a4749ad6568a360d7fa3a08c

However a sequential for loop is, well, sequential. And handling the concurrency by hand would be a pain in the ass (not to mention even if your function is somewhat slow, spawning a thread per element is likely to be way more slow than the gain you get, creating threads is expensive).

I would strongly recommend using rayon, that's exactly the sort of things it's for and it provides a par_iter_mut method which looks to be exactly what you're looking for

vec_of_big_struct.par_iter_mut().for_each(some_slow_function)

or something along those lines should do the trick.

In the same vein, can I pass a range of iterators to a thread ? For exemple I have just two threads, and I want to do one half of the vector done in 1 thread, and the other half in another. — florent teppe, Sep 29 '20 at 10:18
@florentteppe depends what you want exactly: rayon can split an incoming iterable, or you can setup its workers count. If you really want your own threads you'd have to use something like `split_at_mut` to create two mutable slices then crossbeam's scoped threads: with regular threads, as far as the compiler is concerned the threads can "outlive" the current scope, and thus the vector itself, so moving the slices into the workers would not be legal. — Masklinn, Sep 29 '20 at 10:45
Note that rayon also supports *chunking*. That may or may not be what you're looking for. — Masklinn, Sep 29 '20 at 10:46

Concurrently apply the same mutable function to different elements of the same Vec

1 Answers1