Write directly from a reader in Rust

Question

How does one stream data from a reader to a write in Rust?

My end goal is actually to write out some gzipped data in a streaming fashion. It seems like what I am missing is a function to iterate over data from a reader and write it out to a file.

This task would be easy to accomplish with read_to_string, etc. But my requirement is to stream the data to keep memory usage down. I have not been able to find a simple way to do this that doesn't make lots of buffer allocations.

use std::io;
use std::io::prelude::*;
use std::io::{BufReader};
use std::fs::File;
use flate2::read::{GzEncoder};
use flate2::{Compression};

pub fn gzipped<R: Read>(file: String, stream: R) -> io::Result<()> {
    let file = File::create(file)?;
    let gz = BufReader::new(GzEncoder::new(stream, Compression::Default));
    read_write(gz, file)
}

pub fn read_write<R: BufRead, W: Write>(mut r: R, mut w: W) -> io::Result<()> {
  // ?
}

Found a very similar question: [How to read specific number of bytes from a stream](http://stackoverflow.com/questions/30412521/how-to-read-specific-number-of-bytes-from-a-stream). — Matthieu M., Mar 01 '17 at 16:17

Chris Emerson · Accepted Answer · 2017-03-01T16:47:42.390

11

Your read_write function sounds exactly like io::copy. So this would be

pub fn gzipped<R: Read>(file: String, stream: R) -> io::Result<u64> {
    let mut file = File::create(file)?;
    let mut gz = BufReader::new(GzEncoder::new(stream, Compression::Default));
    io::copy(&mut gz, &mut file)
}

The only difference is that io::copy takes mutable references, and returns Result<u64>.

edited Mar 01 '17 at 16:47

answered Mar 01 '17 at 15:03

Chris Emerson

13,041
3
44
66

Thanks! I will see if I can add something to the Rust documentation to make it easier to come across this from iterators and readers and writers. – Greg Weber Mar 01 '17 at 16:46
I think you need &mut for the copy arguments – Greg Weber Mar 01 '17 at 16:46
@GregWeber Thanks - forgot to add them after saying they were needed. – Chris Emerson Mar 01 '17 at 16:48

Write directly from a reader in Rust

1 Answers1