How can I read a file line-by-line, eliminate duplicates, then write back to the same file?

Question

I want to read a file, eliminate all duplicates and write the rest back into the file - like a duplicate cleaner. Vec because a normal array has a fixed size but my .txt is flexible (am I doing this right?).

Read, lines in Vec + delete duplices: Missing write back to file.

use std::io;

fn main() {
    let path = Path::new("test.txt");
    let mut file = io::BufferedReader::new(io::File::open(&path, R));

    let mut lines: Vec<String> = file.lines().map(|x| x.unwrap()).collect();
    // dedup() deletes all duplicates if sort() before
    lines.sort();
    lines.dedup();

    for e in lines.iter() {
        print!("{}", e.as_slice());
    }
}

Read + write to file (untested but should work I guess). Missing lines to Vec because it doesn't work without BufferedReader as it seems (or I'm doing something else wrong, also a good chance).

use std::io;

fn main() {
    let path = Path::new("test.txt");
    let mut file = match io::File::open_mode(&path, io::Open, io::ReadWrite) {
        Ok(f) => f,
        Err(e) => panic!("file error: {}", e),
    };  
    let mut lines: Vec<String> = file.lines().map(|x| x.unwrap()).collect();
    lines.sort();
    // dedup() deletes all duplicates if sort() before
    lines.dedup();

    for e in lines.iter() {
        file.write("{}", e);
    }
}

So .... how do I get those 2 together? :)

Are you interested in algorithm how to do this general or question is about rust specifically? — Jarlax, Jan 10 '15 at 00:33
Both ;). I want to improve my Rust abilites and my regular programer skillset and so I try some real life scripts for fun. — Vega, Jan 10 '15 at 00:50

Shepmaster · Accepted Answer · 2020-07-06T19:37:39.180

Ultimately, you are going to run into a problem: you are trying to write to the same file you are reading from. In this case, it's safe because you are going to read the entire file, so you don't need it after that. However, if you did try to write to the file, you'd see that opening a file for reading doesn't allow writing! Here's the code to do that:

use std::{
    fs::File,
    io::{BufRead, BufReader, Write},
};

fn main() {
    let mut file = File::open("test.txt").expect("file error");
    let reader = BufReader::new(&mut file);

    let mut lines: Vec<_> = reader
        .lines()
        .map(|l| l.expect("Couldn't read a line"))
        .collect();

    lines.sort();
    lines.dedup();

    for line in lines {
        file.write_all(line.as_bytes())
            .expect("Couldn't write to file");
    }
}

Here's the output:

% cat test.txt
    a
    a
    b
    a
                                                                                                                                                                                                                                     % cargo run
thread 'main' panicked at 'Couldn't write to file: Os { code: 9, kind: Other, message: "Bad file descriptor" }', src/main.rs:12:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

You could open the file for both reading and writing:

use std::{
    fs::OpenOptions,
    io::{BufRead, BufReader, Write},
};

fn main() {
    let mut file = OpenOptions::new()
        .read(true)
        .write(true)
        .open("test.txt")
        .expect("file error");

    // Remaining code unchanged
}

But then you'd see that (a) the output is appended and (b) all the newlines are lost on the new lines because BufRead doesn't include them.

We could reset the file pointer back to the beginning, but then you'd probably leave trailing stuff at the end (deduplicating is likely to have less bytes written than read). It's easier to just reopen the file for writing, which will truncate the file. Also, let's use a set data structure to do the deduplication for us!

use std::{
    collections::BTreeSet,
    fs::File,
    io::{BufRead, BufReader, Write},
};

fn main() {
    let file = File::open("test.txt").expect("file error");
    let reader = BufReader::new(file);

    let lines: BTreeSet<_> = reader
        .lines()
        .map(|l| l.expect("Couldn't read a line"))
        .collect();

    let mut file = File::create("test.txt").expect("file error");

    for line in lines {
        file.write_all(line.as_bytes())
            .expect("Couldn't write to file");

        file.write_all(b"\n").expect("Couldn't write to file");
    }
}

And the output:

% cat test.txt
a
a
b
a
a
b
a
b

% cargo run
% cat test.txt
a
b

The less-efficient but shorter solution is to read the entire file as one string and use str::lines:

use std::{
    collections::BTreeSet,
    fs::{self, File},
    io::Write,
};

fn main() {
    let contents = fs::read_to_string("test.txt").expect("can't read");
    let lines: BTreeSet<_> = contents.lines().collect();

    let mut file = File::open("test.txt").expect("can't create");
    for line in lines {
        writeln!(file, "{}", line).expect("can't write");
    }
}

How can I read a file line-by-line, eliminate duplicates, then write back to the same file?

1 Answers1

Linked