S3 Upload Using Rusoto

Question

I'm a rust noob. Have been trying to upload a simple text file to S3 using rusoto but haven't been able to convert a file into Stream. How to convert a file into a Stream acceptable by rusoto S3 upload.

let s3_client = S3Client::new(Region::UsEast1);
let file = File::open("text.txt").await?;
let mut reader = BufReader::new(file);
let mut buffer: Stream = Vec::new();
file.read_to_end(&mut buffer).await?;

let result = s3_client.put_object(PutObjectRequest {
    bucket: String::from("bucket-name"),
    key: "text.txt".to_string(),
    body: Some(StreamingBody::new(buffer)),
    ..Default::default()
});

Lytigas · Accepted Answer · 2020-08-24T01:24:08.373

The StreamingBody type used in the S3 client is an alias for ByteStream from rusoto_core, which can be made either from a type implementing futures_core::stream::Stream or from a Vec<u8>.

The most correct way to accomplish this would be to use a async read of the file via Tokyo, and then create a ByteStream from the resulting Stream of bytes. But reading a file as a Stream doesn't have a fully fleshed out solution in the async ecosystem at the moment. See this SO question, especially the links at the bottom of the accepted answer, for more details.

Another way is to read the file fully into memory, as you are doing in the provided code example, and then create a ByteStream from the resulting Vec<u8>. If you do that your code is pretty close.

use tokio::io::AsyncReadExt;
use rusoto_s3::*;
use rusoto_core::*;

async fn do_upload() -> Result<(), Box<dyn std::error::Error>> {
    let s3_client = S3Client::new(Region::UsEast1);
    let mut file = tokio::fs::File::open("text.txt").await?;
    let mut buffer = Vec::new();
    file.read_to_end(&mut buffer).await?;

    let result = s3_client.put_object(PutObjectRequest {
        bucket: String::from("bucket-name"),
        key: "text.txt".to_string(),
        body: Some(StreamingBody::from(buffer)),
        ..Default::default()
    }).await?;
    // do thing with result
    Ok(())
}

The only real changes are:

removing the Stream annotation from buffer, which doesn't really make sense because Stream is a trait and not a concrete type. Also, we can't use it easily for the aforementioned reasons.
StreamingBody::from rather than ::new, because we are using a Vec and not a type implementing Stream.

S3 Upload Using Rusoto

1 Answers1