Value does not live long enough when using the Chunks iterator across multiple threads

Question

This is a simplified example of my situation:

use std::sync::mpsc::{Sender, Receiver};
use std::sync::mpsc;
use std::thread;

struct User {
    reference: String,
    email: String,
}

fn main() {
    let rows = vec![
        vec!["abcd", "test@test.com"],
        vec!["efgh", "test1@test.com"],
        vec!["wfee", "test2@test.com"],
        vec!["rrgr", "test3@test.com"],
    ];

    let chunk_len = (rows.len() / 2) as usize;
    let mut chunks = Vec::new();
    for chunk in rows.chunks(chunk_len) {
        chunks.push(chunk);
    }

    let (tx, rx): (Sender<Vec<User>>, Receiver<Vec<User>>) = mpsc::channel();

    for chunk in chunks {
        let thread_tx = tx.clone();
        thread::spawn(move || {
            let result = chunk
                .iter()
                .map(|row| {
                    User {
                        reference: row[0].to_string(),
                        email: row[1].to_string(),
                    }
                })
                .collect::<Vec<User>>();
            thread_tx.send(result).unwrap();
        });
    }

    let mut users = Vec::new();
    for _ in 0..chunk_len {
        users.push(rx.recv());
    }
}

and it's throwing an error

error[E0597]: `rows` does not live long enough
  --> src/main.rs:20:18
   |
20 |     for chunk in rows.chunks(chunk_len) {
   |                  ^^^^ does not live long enough
...
47 | }
   | - borrowed value only lives until here
   |
   = note: borrowed value must be valid for the static lifetime...

I tried to change to chunks.push(chunk.clone()); but the error still wouldn't go away. What am I missing here?

@Shepmaster: While the error message is pretty generic, I think the fundamental issue here is actually a mis-understanding of what `chunks` creates. `chunks` (and its clones) are references into the vector while the OP seems to believe they are stand-alone. — Matthieu M., Jun 15 '15 at 13:32

score 6 · Accepted Answer · edited Jul 25 '17 at 23:06

The error does look misleading, however, it is correct. Your problem in fact is that chunks() gives an iterator of slices into the original vector:

impl<'a, T> Iterator for Chunks<'a, T> {
    type Item = &'a [T];
}

You're trying to use this slice in a spawn()ed thread which requires the closure to have the 'static lifetime:

pub fn spawn<F, T>(f: F) -> JoinHandle<T> 
where
    F: FnOnce() -> T,
    F: Send + 'static,
    T: Send + 'static,

Your iterator does have the 'static lifetime because it contains references to a runtime-allocated vector.

You said that you tried clone(), but that only clones the slice, it does not give you a new vector. For that you need to use to_owned():

for chunk in rows.chunks(chunk_len) {
    chunks.push(chunk.to_owned());
}

And lo, it compiles.

Or just `let chunks: Vec<_> = rows.chunks(chunk_len).map(|c| c.to_owned()).collect();` — Shepmaster, Jul 25 '17 at 23:12

score 3 · Answer 2 · edited Jul 25 '17 at 23:11

The problem you are experiencing is that std::slice::Chunks is an iterator that returns references to the original vector. Therefore, in

let mut chunks = Vec::new();
for chunk in rows.chunks(chunk_len) {
    chunks.push(chunk);
}

chunks ends up being a vector of references into rows (or to be exact a collections::vec::Vec<&[collections::vec::Vec<&str>]>).

You would need to create a new Vec inside this loop, which would copy the data. Unfortunately, calling clone is insufficient (it creates a clone of the reference, which is still a reference). You could do:

for chunk in rows.chunks(chunk_len) {
    let mut v = Vec::new();
    v.extend(chunk.iter().cloned());
    chunks.push(v);
}

EDIT: Vladimir has a much better way to clone the slice => chunks.push(chunk.to_owned()).

score 0 · Answer 3 · answered Nov 22 '22 at 07:45

Another option is to use the scoped_threadpool crate, which allows threads to borrow non-'static references:

use scoped_threadpool::Pool; // 0.1.9
Pool::new(2).scoped(|scope| {
    for chunk in chunks {
        let thread_tx = tx.clone();
        scope.execute(move || {
            let result = chunk
                .iter()
                .map(|row| User {
                    reference: row[0].to_string(),
                    email: row[1].to_string(),
                })
                .collect::<Vec<User>>();
            thread_tx.send(result).unwrap();
        });
    }
});

Playground

Value does not live long enough when using the Chunks iterator across multiple threads

3 Answers3

Linked

Related