"cannot infer an appropriate lifetime" when attempting to return a chunked response with hyper

Question

I would like to return binary data in chunks of specific size. Here is a minimal example.

I made a wrapper struct for hyper::Response to hold my data like status, status text, headers and the resource to return:

pub struct Response<'a> {
    pub resource: Option<&'a Resource>
}

This struct has a build method that creates the hyper::Response:

impl<'a> Response<'a> {
    pub fn build(&mut self) -> Result<hyper::Response<hyper::Body>, hyper::http::Error> {
        let mut response = hyper::Response::builder();
        match self.resource {
            Some(r) => {
                let chunks = r.data
                .chunks(100)
                .map(Result::<_, std::convert::Infallible>::Ok);
                response.body(hyper::Body::wrap_stream(stream::iter(chunks)))       
            },
            None => response.body(hyper::Body::from("")),
        }
    }    
}

There is also another struct holding the database content:

pub struct Resource {
    pub data: Vec<u8>
}

Everything works until I try to create a chunked response. The Rust compiler gives me the following error:

error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
  --> src/main.rs:14:15
   |
14 |         match self.resource {
   |               ^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'a` as defined on the impl at 11:6...
  --> src/main.rs:11:6
   |
11 | impl<'a> Response<'a> {
   |      ^^
note: ...so that the types are compatible
  --> src/main.rs:14:15
   |
14 |         match self.resource {
   |               ^^^^^^^^^^^^^
   = note: expected `Option<&Resource>`
              found `Option<&'a Resource>`
   = note: but, the lifetime must be valid for the static lifetime...
note: ...so that the types are compatible
  --> src/main.rs:19:31
   |
19 |                 response.body(hyper::Body::wrap_stream(stream::iter(chunks)))       
   |                               ^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `From<&[u8]>`
              found `From<&'static [u8]>`

I don't know how to fulfill these lifetime requirements. How can I do this correctly?

Answer 1

The problem is not in the 'a itself, but in the fact that the std::slice::chunks() function returns an iterator that borrows the original slice. You are trying to create a stream future from this Chunks<'_, u8> value, but the stream requires it to be 'static. Even if your Resource did not have the 'a lifetime, you would still have the r.data borrowed, and it would still fail.

Remember that here 'static does not mean that the value lives forever, but that it can be made to live as long as necessary. That is, the future must not hold any (non-'static) borrows.

You could clone all the data, but if it is very big, it can be costly. If so, you could try using Bytes, that is just like Vec<u8> but reference counted.

It looks like there is no Bytes::chunks() function that returns an iterator of Bytes. Fortunately it is easy to do it by hand.

Lastly, remember that iterators in Rust are lazy, so they keep the original data borrowed, even if it is a Bytes. So we need to collect them into a Vec to actually own the data (playground):

pub struct Resource {
    pub data: Bytes,
}

impl<'a> Response<'a> {
    pub fn build(&mut self) -> Result<hyper::Response<hyper::Body>, hyper::http::Error> {
        let mut response = hyper::Response::builder();
        match self.resource {
            Some(r) => {
                let len = r.data.len();
                let chunks = (0..len)
                    .step_by(100)
                    .map(|x| {
                        let range = x..len.min(x + 100);
                        Ok(r.data.slice(range))
                    })
                    .collect::<Vec<Result<Bytes, std::convert::Infallible>>>();
                response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
            }
            None => response.body(hyper::Body::from("")),
        }
    }
}

UPDATE: We can avoid the call to collect() if we notice that stream::iter() takes ownership of an IntoIterator that can be evaluated lazily, as long as we make it 'static. It can be done if we do a (cheap) clone of r.data and move it into the lambda (playground):

    let data = r.data.clone();
    let len = data.len();
    let chunks = (0..len).step_by(100)
        .map(move |x| {
            let range = x .. len.min(x + 100);
            Result::<_, std::convert::Infallible>::Ok(data.slice(range))
                    });
    response.body(hyper::Body::wrap_stream(stream::iter(chunks)))