Create an iterator and put it into a new struct without bothering the borrow-checker

Question

I am trying to create a lexical analyzer which uses itertools::PutBack to make an iterator over the characters in a String. I intend to store the pushback iterator in a struct and delegate methods to it so that I can categorize the characters by an enum, which will then be passed to a state machine at the core of the lexical analyzer (not yet written).

The borrow-checker is not happy with me. Method ParserEventIterator::new near the bottom of the listing causes the error. How do I define the lifetimes or borrowing so that I can get this to compile? Or what Rustic data structure design should I use in its stead?

Ultimately, I would like this to implement the appropriate traits to become a proper iterator. (Newbie to Rust. Prior to this, I have programmed in 28 languages, but this one has me stumped.)

Here is a code sample:

extern crate itertools;
use itertools::put_back;
use std::fmt::Display;
use std::fmt::Formatter;
use std::fmt::Result;

pub enum ParserEvent {
    Letter(char),
    Digit(char),
    Other(char),
}

impl ParserEvent {
    fn new(c: char) -> ParserEvent {
        match c {
            'a'...'z' | 'A'...'Z' => ParserEvent::Letter(c),
            '0'...'9' => ParserEvent::Digit(c),
            _ => ParserEvent::Other(c),
        }
    }
}

impl Display for ParserEvent {
    fn fmt(&self, f: &mut Formatter) -> Result {
        let mut _ctos = |c: char| write!(f, "{}", c.to_string());
        match self {
            ParserEvent::Letter(letter) => _ctos(*letter),
            ParserEvent::Digit(digit) => _ctos(*digit),
            ParserEvent::Other(o) => _ctos(*o),
        }
    }
}

//  ParserEventIterator
//  Elements ('e) must have lifetime longer than the iterator ('i).
pub struct ParserEventIterator<'i, 'e: 'i> {
    char_iter: &'i mut itertools::PutBack<std::str::Chars<'e>>,
}

impl<'i, 'e: 'i> ParserEventIterator<'i, 'e> {
    fn new(s: &'e std::string::String) -> ParserEventIterator<'i, 'e> {
        // THIS NEXT LINE IS THE LINE WITH THE PROBLEM!!!
        ParserEventIterator {
            char_iter: &mut put_back(s.chars()),
        }
    }

    fn put_back(&mut self, e: ParserEvent) -> () {
        if let Some(c) = e.to_string().chars().next() {
            self.char_iter.put_back(c);
        }
    }
}

impl<'i, 'e: 'i> Iterator for ParserEventIterator<'i, 'e> {
    type Item = ParserEvent;
    fn next(&mut self) -> Option<ParserEvent> {
        match self.char_iter.next() {
            Some(c) => Some(ParserEvent::new(c)),
            None => None,
        }
    }
}

fn main() {
    let mut _i = ParserEventIterator::new(&String::from("Hello World"));
}

On the Rust Playground

error[E0515]: cannot return value referencing temporary value
  --> src/main.rs:43:9
   |
43 | /         ParserEventIterator {
44 | |             char_iter: &mut put_back(s.chars()),
   | |                             ------------------- temporary value created here
45 | |         }
   | |_________^ returns a value referencing data owned by the current function

Peter Varo · Accepted Answer · 2019-05-21T21:36:38.677

1

Well, the compiler is almost telling you the solution by reflecting to the obvious problem: you can't have a borrow which doesn't live long enough, i.e. the borrow would point to a nonexistent location after the stack memory of the function has been destroyed.

This would happen because the borrow is referencing an object (in this case an itertools::struct::PutBack instance) that has been newly created within the function body. This instance gets destroyed at the end of the function along with all the references to it. So the compiler is preventing you to have a so called dangling pointer.

Thus, instead of borrowing you should move the PutBack instance into your struct:

// ...

pub struct ParserEventIterator<'e> {
    char_iter: itertools::PutBack<std::str::Chars<'e>>
}

impl<'e> ParserEventIterator<'e> {
    fn new(s: &'e std::string::String) -> ParserEventIterator<'e> {
        ParserEventIterator { char_iter: put_back(s.chars()) }
    }

    // ...
}

edited May 21 '19 at 21:36

answered May 21 '19 at 21:26

Peter Varo

11,726
7
55
77

What you suggest was in fact my starting point. It yields this error: | expected mutable reference, found struct `itertools::PutBack` | help: consider mutably borrowing here: `&mut put_back(s.chars() }` – Paul Chernoch May 22 '19 at 13:55
Then if I follow that advice, i get `returns a value referencing data owned by the current function` – Paul Chernoch May 22 '19 at 13:58
I didn't notice your changes to the lifetimes. Testing it again. – Paul Chernoch May 22 '19 at 14:32
1

Your proposal compiles! On to the next problem... – Paul Chernoch May 22 '19 at 14:34
The devil is always in the detail, isn't he? Anyway, I'm glad if I could help ;) – Peter Varo May 22 '19 at 15:11

Create an iterator and put it into a new struct without bothering the borrow-checker

1 Answers1