1

I'm trying to implement the Monkey toy language in Rust. I'm now trying to generate and AST, but before that, I need to have a functioning parser. I already have my lexer.

So the relevant part of my parser looks like this:

struct Parser<'a> {
    lexer: lexer::Lexer<'a>,
    current_token: Token<'a>,
    peek_token: Token<'a>,
}

impl<'a> Parser<'a> {
    // Create a new parser.
    // Depends on a lexer capable of iterating over
    // the tokens.
    pub fn new(mut lexer: lexer::Lexer<'a>) -> Parser {
        // constructor
    }

    // Read the next token.
    //
    // Returns self to avoid weird `move out of
    // borrowed content` issue?
    fn next_token(mut self) -> Self {
        self.current_token = self.peek_token;
        self.peek_token = self.lexer.next_token();
        self
    }
}

The problem is in the next_token function. I would really like it to work with a borrowed &self, like this:

// Read the next token.
fn next_token(&mut self) {
    self.current_token = self.peek_token;
    self.peek_token = self.lexer.next_token();
}

But the compiler complaints about moving out of a borrowed content. Now, I understand why this is happening: I'm only borrowing self, I cannot perform the move from self.peek_token to self.current_token because I do not own it.

My question though, is, is returning Self the best strategy? The code which returns Self works fine, but the interface just got very very ugly.

#[test]
fn test_next_token() {
    let l = lexer::Lexer::new("let answer = 42;");
    let p = Parser::new(l);
    assert_eq!(p.current_token, Token::Let);
    let p = p.next_token();
    assert_eq!(p.current_token, Token::Ident("answer"));
}

Is there an alternative I'm not seeing? Is this a common idiom in Rust?

Here is a link to the playground.

hellow
  • 12,430
  • 7
  • 56
  • 79
joaquinlpereyra
  • 956
  • 7
  • 17

1 Answers1

3

You can use std::mem::replace:

use std::mem;

// Read the next token.
fn next_token(&mut self) {
    self.current_token = mem::replace(&mut self.peek_token, self.lexer.next_token());
}

replace will store the second argument into the first, and return the current value of the first, which we then assign to current_token.

harmic
  • 28,606
  • 5
  • 67
  • 91
  • Nice! That worked. Seeing your answer though I do not think I understood so clearly why I couldn't just move out of the variable before? I rechecked the borrowing rules and there's nothing saying that you cannot move out of borrowed content? Obviously in my original code rust would invalidate `self.peek_token` after the move, as it could lead to a user after free, right? But why doesn't it just do that instead of saying I can't do it? Only reason I came up with is that there may be another thread expecting `self.peek_token` to be valid? – joaquinlpereyra Apr 16 '19 at 05:17
  • 3
    Simply put, because `panic!`. You move out of `peek_token`, invalidating it and expecting to make it valid again with the result of `next_token()`. But if `next_token()` instead panics, and some outer code catches that panic, you have a view of the struct with an invalidated field. (Threading, by the way, is not a concern, because a `&mut` is exclusive and guarantees that no other thread is currently accessing the struct.) By using `replace`, you guarantee that no panics can occur between invalidation and reinitialization. – Sebastian Redl Apr 16 '19 at 06:22