Writing a chunk stream to a file asynchronously using hyper

Question

I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs for that). Here is the code:

View in the playground

// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
    uri: Uri,
    file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> {
    let temp_dir_path = tempfile::tempdir().unwrap().into_path();
    let file_name = match file_location.file_name() {
        Some(file_name) => file_name,
        None => return Box::new(futures::failed(DownloadFileError::IncorrectFilePath)),
    };

    let temp_filepath = temp_dir_path.join(&file_name);

    let connector = HttpsConnector::new(2).unwrap();
    let client: Client<_, Body> = Client::builder().build(connector);

    let response_future = client
        .get(uri)
        .map_err(|err| DownloadFileError::GetRequest(err));

    let create_file_future =
        File::create(temp_filepath).map_err(|err| DownloadFileError::CreateFile(err));

    Box::new(
        response_future
            .join(create_file_future)
            .and_then(move |(res, file)| {
                res.into_body()
                    .map_err(|e| DownloadFileError::GetRequest(e))
                    .for_each(move |chunk| {
                        io::write_all(file, chunk)
                            .map(|_| ())
                            .map_err(|_| DownloadFileError::FileWrite)
                    })
            }),
    )
}

However, I get the following error:

error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
  --> src/lib.rs:79:39
   |
75 |             .and_then(move |(res, file)| {
   |                                   ---- captured outer variable
...
79 |                         io::write_all(file, chunk)
   |                                       ^^^^ cannot move out of captured outer variable in an `FnMut` closure

Conceptually, I understand what the error means: Since a FnMut captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join future.

The write_all method from the Write trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.

I think there is no point to use `for_each()`, `into_body()` return an option so just use `map()` instead. But maybe I totally wrong I don't understand a thing in hyper/futures — Stargateur, Nov 16 '18 at 02:45
As far as I understand from the answers both of them write synchronously to the file using `write_all`. In my case `io::write_all` is done asynchronously as well. — lhahn, Nov 16 '18 at 03:13

Shepmaster · Accepted Answer · 2018-11-17T03:41:21.713

You do not want to use for_each. io::write_all consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold to reuse the file:

.fold(file, |file, chunk| {
    io::write_all(file, chunk)
        .map(|(f, _c)| f)
        .map_err(|_| DownloadFileError::FileWrite)
})
.map(drop)

Writing a chunk stream to a file asynchronously using hyper

1 Answers1

Linked