Rust borrowing compilation errors

Question

I'm trying to learn Rust by writing a simple lexer. Here is what I have so far...

use std::fs::File;
use std::io::Read;
use std::str::Chars;

pub struct Lexer<'a> {
    file_name: String,
    file_contents: String,
    iterator: Option<Chars<'a>>,
}

impl<'a> Lexer<'a> {

    fn new(fname: &str) -> Lexer {
        Lexer {
            file_name: fname.to_string(),
            file_contents: String::new(),
            iterator: None,
        }
    }


    // Reads the file contents and creates iterator
    fn init(&'a mut self) { 

        // Open the file
        let mut f = File::open(&self.file_name).expect("Couldn't open file");

        // Read the contents
        f.read_to_string(&mut self.file_contents).expect("Couldn't read file contents");

        self.iterator = Some(self.file_contents.chars());

    }

    // Gets the next character
    fn get_next(&mut self) -> Option<char> {

        self.iterator.unwrap().next()

    }

}

fn main() {

    let mut lexer = Lexer::new("test.txt");
    lexer.init();

    // Assuming the file "text.txt" contains "Hello World" 
    // then the following two lines should print "H" then "e"

    println!("{}", lexer.get_next().unwrap());
    println!("{}", lexer.get_next().unwrap());

}

However when I try to compile it I get the following two errors:

cannot move out of borrowed content [E0507]
main.rs:38          self.iterator.unwrap().next()

and

cannot borrow `lexer` as mutable more than once at a time [E0499]
main.rs:49      println!("{}", lexer.get_next().unwrap());

A Google of the first error reveals that Clone()-ing is a possible solution for this kind of error, but I believe this won't work in this case as the iterators state needs to update every time next() is called.

Does anyone have any suggestions on how to overcome these issues and get it to compile?

Answer 1

Ultimately, you are attempting to store a value and a reference to that value in the same struct. Unlike other formulations, this specific case allows you to "tie the knot" of references, but probably doesn't do what you want. For example, after calling init, you will never be able to move the Lexer because moving it would invalidate the reference.

It also explains the "borrow again" error. Because the lifetime is applied to self, and it's a mutable reference, the lexer itself will keep the mutable reference forever, meaning nothing else can ever mutate it, including itself.

The short answer is: don't organize your code that way. For whatever reason, parsing and lexing is a popular problem in the Rust community. Check out how other libraries do it.

Or check out how iterators work in general. The item being iterated over stays in place, and a separate iterator is returned that references the original item.

Splitting your code into the same two pieces will likely be the best direction.