-1

I am trying to take a large string or data and break it up into slices to process across multiple thread. I would like to leverage multiple thread to help process the data but want to avoid copying data.

Here is an example of the problem. Segment0 will work because it owns the data through a copy. Segment1 just borrows the data and will cause a problem even though the joins guarantee lifetime correctness.

use std::thread;
use std::fs;
use std::sync::{Arc};

fn main() {
    let data = fs::read("./test.txt").unwrap();

    let segment_size = data.len()/2;
    let segment0 = &data[0..segment_size]; //Borrow data no copy
    let segment1 = &data[segment_size..]; //Borrow data no copy

    let segment0 = Vec::from(segment0); //has ownership of data through copy

    let thread0 = thread::spawn(move || println!("Thread 0 {:?}",std::str::from_utf8(&segment0)));
    let thread1 = thread::spawn(move || println!("Thread 1 {:?}",std::str::from_utf8(segment1)));

    thread0.join(); //thread 0 terminated segment0 dropped
    thread1.join(); //thread 1 terminated segment1 dropped

    //data released  segment0 and segment1 needed to be dropped before
}

The error returned

error[E0597]: `data` does not live long enough
--> src/main.rs:10:25
|
10 |         let segment1 = &data[segment_size..]; //Borrow data no copy
|                         ^^^^ borrowed value does not live long enough
...
21 |     }
|     - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
Justin Bullard
  • 33
  • 3

1 Answers1

0

Thank you scoped_threadpool solved my problem here is my solution

extern crate scoped_threadpool;
use std::thread;
use std::fs;
use std::sync::{Arc};

use scoped_threadpool::Pool;


fn main() {
    let data = fs::read("./test.txt").unwrap();

    let segment_size = data.len() / 2;
    let segment0 = &data[0..segment_size]; //Borrow data no copy
    let segment1 = &data[segment_size..]; //Borrow data no copy

    let mut pool = Pool::new(2);

    pool.scoped(|scoped| {
        scoped.execute(move || println!("Thread 0 {:?}", std::str::from_utf8(segment0)));
        scoped.execute(move || println!("Thread 1 {:?}", std::str::from_utf8(segment1)));
    });
}
Justin Bullard
  • 33
  • 3
  • The scoped-thread-pool introducing a lot of overhead for something so basic. It would be great if the rust lifetimes could understand thread scope lifetimes along with just code scope. – Justin Bullard May 27 '18 at 22:13
  • 5
    The Rust thread model used to support this, with an intelligent `JoinGuard` type that would clean up when it went out of scope; but it was found to be unsafe (and not easy to make safe). The problem is described in [this creatively named article](http://cglab.ca/~abeinges/blah/everyone-poops/), which I recommend; it changed my perspective on the issue dramatically. – trent May 27 '18 at 22:53
  • 1
    Note that you did not have to answer this question: as a duplicate question, the answers to it are already available in its respective target. – E_net4 May 28 '18 at 08:13