How to iterate over stdin twice?

Question

I need to create two iterators for stdin. However, I cannot figure out how to implement this.

Here is a simple example:

use std::io::{self, BufRead, BufReader, Result};

fn main() {
    let reader1 = BufReader::new(io::stdin()).lines();
    let reader2 = BufReader::new(io::stdin()).lines();

    for line in reader1 {
        println!("reader1: {:?}", line);
    }

    for line in reader2 {
        println!("reader2: {:?}", line);
    }
}

output:

$ printf '1\n2\n3\n4\n' | cargo run
reader1: Ok("1")
reader1: Ok("2")
reader1: Ok("3")
reader1: Ok("4")

The iterator reader2 never gets implemented.

The full example of what I am trying to accomplish is a little more complex...

use itertools::multipeek;
use std::io::{self, BufRead, BufReader, Result};

fn main() {
    let reader = BufReader::new(io::stdin()).lines();
    let mut mp = multipeek(BufReader::new(io::stdin()).lines());

    for line in reader {
        mp.next();
        match line {
            Ok(l) => {
                println!("line: {}", l);
                println!("peek: {:?}", mp.peek());
                println!("peek: {:?}", mp.peek());
            }
            Err(e) => println!("error parsing line: {:?}", e),
        }
    }
}

output:

$ printf '1\n2\n3\n4\n' | cargo run
line: 1
peek: None
peek: None
line: 2
peek: None
peek: None
line: 3
peek: None
peek: None
line: 4
peek: None
peek: None

from what I can tell though, it has something to do with using stdin because it works fine when reading from a file.

Answer 1

I am not familiar with Rust enough to offer a concrete solution, but you are failing because on lines 6-8 you process stdin (into lines), then on lines 10-12 you try to process it again. But there is nothing left to process a second time; you processed it already.

Remember, reader1 and reader2 are readers, not buffers. They do not duplicate the input stream.

If you wish to review any part of prior input, you must read it into some kind of storage object. Then you can create a BufReader or otherwise prepare access to that data.

From what you appear to be doing, an easy way would be simply to create a single BufReader object to populate an array of strings with the input. Then just iterate over that array as many times as you want.

Answer 2

Alternative solution to the second problem that avoids duplicating the stdin iterator and allows for look-ahead with itertools::multipeek

use std::io::{self, BufRead, BufReader, Result};
use itertools::multipeek;

fn main(){
    let num_peeks = 2;
    let mut mp = multipeek(BufReader::new(io::stdin()).lines());
    loop {
        let a = mp.next();
        match a {
            Some(l) => {
                println!("line: {:?}", l);

                for _ in 0..num_peeks {
                    println!("peek: {:?}", mp.peek());
                }
            }
            None => break,
        }
    }
}

output:

$ printf '1\n2\n3\n4\n' | cargo run
line: Ok("1")
peek: Some(Ok("2"))
peek: Some(Ok("3"))
line: Ok("2")
peek: Some(Ok("3"))
peek: Some(Ok("4"))
line: Ok("3")
peek: Some(Ok("4"))
peek: None
line: Ok("4")
peek: None
peek: None

Answer 3

You can't iterate over stdin twice, because you can read it once only. If you want to do something similar, you need to buffer the data in your app yourself.

In your question, you duplicate not only the stream, but also create two buffered readers. This is important, because the buffered reader will likely read more data than required each time. For example, to get lines of text, it won't read byte-after-byte looking for a newline because that would be slow. Most likely it will read a multiple-of-page-size blocks into a preallocated space instead. Then your lines will come from there without any stream reads until it runs out of lines to return.

Your solution works because you have only one backing buffer, so everything works as expected. In the question though, after the first execution of for line in reader you don't have anything in your stream anymore - it's all been consumed. So mp.next() / mp.peek() won't read anything.

If you want to confirm this, try with input of multiple KB - you'll see mp.peek return values from the middle of your input.