3

I want to download a file from a URL. The code below works fine, but when I replace Vec with [u8; 4 * 1024] and read_to_end with read within the loop, I get infinite reading of the same information. How should I change my code to get the file downloaded in parts using a fixed-size array?

use std::env;
use std::fs::{File, OpenOptions};
use std::io::{BufRead, BufReader, BufWriter, Read, Result, Write};

//[dependencies]
//reqwest = "0.8.0"
extern crate reqwest;

fn main() -> Result<()> {
    // All URL's are stored in file passed as a command line argument
    let args: Vec<String> = env::args().collect();
    let fin = File::open(&args[1])?;

    let buf: &mut Vec<u8> = &mut Vec::new();
    let mut readed: usize;
    for line in BufReader::new(fin).lines() {
        let unwrapped = &line.unwrap();
        let mut options = OpenOptions::new();

        // Create an output file with a name equal to the last domain
        let fout = options
            .write(true)
            .create(true)
            .open(&unwrapped[unwrapped.rfind('/').unwrap() + 1..])?;
        let mut writer = BufWriter::new(&fout);

        readed = reqwest::get(unwrapped).unwrap().read_to_end(buf).unwrap();
        println!("{}", readed); // Just to know how many read
        writer.write(buf)?;

        // Close the file and clear "buf" to ensure that
        // the subsequent reading is correct
        writer.flush()?;
        buf.clear();
    }
    Ok(())
}
Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
k0shinus
  • 81
  • 1
  • 6
  • Have you seen [How to idiomatically / efficiently pipe data from Read+Seek to Write?](https://stackoverflow.com/q/33461772/155423) – Shepmaster Sep 12 '18 at 19:59
  • What you say don't make sense, you use read_to_end on the result of your get on yours URLs, that nothing to do with your file. – Stargateur Sep 12 '18 at 20:02
  • @Shepmaster It doesn't look like `io::copy` or `io::SeekFrom` can work with net files. I am wrong? – k0shinus Sep 12 '18 at 20:13
  • @Stargateur But it works almost exactly as i want: i gave the URL's and got the correct files on my computer. – k0shinus Sep 12 '18 at 20:27
  • 2
    *It doesn't look like io::copy or io::SeekFrom can work with net files. I am wrong?* — yes, [I believe you are wrong](https://play.rust-lang.org/?gist=c97a4bb7c66a03a6f84597bce7e6c033&version=stable&mode=debug&edition=2015). – Shepmaster Sep 12 '18 at 20:41
  • @Shepmaster `reqwest::get` get the full net file content and then write it in parts with `io::copy` to file on my machine? – k0shinus Sep 13 '18 at 13:27
  • I would expect that it will transfer the file from the remote server in chunks (including OS socket buffers and probably some framework buffers) and write those chunks to a file, yes. For example, if you downloaded a file that was 1 GB, I would not expect more than a few MB of RAM to be used at a time. – Shepmaster Sep 13 '18 at 13:33

0 Answers0