4

I'm building a parser using nom and I'd like to use nom combinators to process tokenized input.

The stream of tokens is represented by Vec<Token<S>>, with Token being:

#[derive(Debug, Clone, PartialEq)]
pub enum Token<S> {
    Identifier(S),
    String(S),
}

I tried to implement InputIter for a vector of arbitrary values:

impl<'a, T> InputIter for Vec<T> {
    type Item = T;
    type Iter = Enumerate<Self::IterElem>;
    type IterElem = Iter<'a, Self::Item>;

    fn iter_indices(&self) -> Self::Iter {
        self.iter().enumerate()
    }

    fn iter_elements(&self) -> Self::IterElem {
        self.iter()
    }
}

When I encountered the following error:

error[E0207]: the lifetime parameter `'a` is not constrained by the impl trait, self type, or predicates
  --> src/lex/tokens.rs:37:6
   |
37 | impl<'a, T> InputIter for Vec<T> {
   |      ^^ unconstrained lifetime parameter

I then tried to constrain the lifetime parameter using the type parameter Token<S>:

impl<'a> InputIter for Vec<Token<&'a str>> {
    type Item = Token<&'a str>;
    type Iter = Enumerate<Self::IterElem>;
    type IterElem = Iter<'a, Self::Item>;

    fn iter_indices(&self) -> Self::Iter {
        self.iter().enumerate()
    }

    fn iter_elements(&self) -> Self::IterElem {
        self.iter()
    }
}

Which led to the following:

error[E0117]: only traits defined in the current crate can be implemented for arbitrary types
  --> src/lex/tokens.rs:51:1
   |
51 | impl<'a> InputIter for Vec<Token<&'a str>> {
   | ^^^^^^^^^^^^^^^^^^^^^^^-------------------
   | |                      |
   | |                      `std::vec::Vec` is not defined in the current crate
   | impl doesn't use only types from inside the current crate
   |
   = note: define and implement a trait or new type instead

The error seems to imply that I can't implement an outside trait for an outside type, so I'm trying to wrap Vec<Token<S>> into a custom struct:

struct Tokens<S> {
    vec: Vec<Token<S>>,
}

impl<'a> InputIter for Tokens<&'a str> {
    type Item = Token<&'a str>;
    type Iter = Enumerate<Self::IterElem>;
    type IterElem = Iter<'a, Self::Item>;

    fn iter_indices(&self) -> Self::Iter {
        self.vec.iter().enumerate()
    }

    fn iter_elements(&self) -> Self::IterElem {
        self.vec.iter()
    }
}

Which results in the following error:

error[E0271]: type mismatch resolving `<std::iter::Enumerate<std::slice::Iter<'_, lex::tokens::Token<&str>>> as std::iter::Iterator>::Item == (usize, lex::tokens::Token<&str>)`
  --> src/lex/tokens.rs:56:3
   |
55 | impl<'a> InputIter for Tokens<&'a str> {
   | -------------------------------------- in this `impl` item
56 |   type Item = Token<&'a str>;
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found enum `lex::tokens::Token`
   |
   = note: expected tuple `(_, &lex::tokens::Token<&str>)`
              found tuple `(_, lex::tokens::Token<&str>)`

Does this mean I'm restricted to using a vector of references as opposed to a vector containing tokens directly? What gives? There doesn't seem to be anything in the InputIter declaration to imply this constraint.

Is there another way to accomplish what I'm trying to do and represent token stream in a way that is compatible with nom combinators?

A reproducible example playground:

use std::iter::Enumerate;
use std::slice::Iter;

#[derive(Debug, Clone, PartialEq)]
pub enum Token<S> {
    Identifier(S),
    String(S),
}

struct Tokens<S> {
    vec: Vec<Token<S>>,
}

pub trait InputIter {
    type Item;
    type Iter: Iterator<Item = (usize, Self::Item)>;
    type IterElem: Iterator<Item = Self::Item>;

    fn iter_indices(&self) -> Self::Iter;
    fn iter_elements(&self) -> Self::IterElem;
}

impl<'a> InputIter for Tokens<&'a str> {
    type Item = Token<&'a str>;
    type Iter = Enumerate<Self::IterElem>;
    type IterElem = Iter<'a, Token<&'a str>>;

    fn iter_indices(&self) -> Self::Iter {
        self.vec.iter().enumerate()
    }

    fn iter_elements(&self) -> Self::IterElem {
        self.vec.iter()
    }
}

fn main() {}
error[E0271]: type mismatch resolving `<std::slice::Iter<'_, Token<&str>> as std::iter::Iterator>::Item == Token<&str>`
  --> src/main.rs:24:5
   |
23 | impl<'a> InputIter for Tokens<&'a str> {
   | -------------------------------------- in this `impl` item
24 |     type Item = Token<&'a str>;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found enum `Token`
   |
   = note: expected reference `&Token<&str>`
                   found enum `Token<&str>`

error[E0271]: type mismatch resolving `<std::iter::Enumerate<std::slice::Iter<'_, Token<&str>>> as std::iter::Iterator>::Item == (usize, Token<&str>)`
  --> src/main.rs:24:5
   |
23 | impl<'a> InputIter for Tokens<&'a str> {
   | -------------------------------------- in this `impl` item
24 |     type Item = Token<&'a str>;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found enum `Token`
   |
   = note: expected tuple `(_, &Token<&str>)`
              found tuple `(_, Token<&str>)`
Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
mpontus
  • 2,143
  • 2
  • 19
  • 21
  • @Stargateur Updated with a link to the playground. – mpontus Apr 13 '20 at 15:12
  • It looks like your question might be answered by the answers of [What is the difference between iter and into_iter?](https://stackoverflow.com/q/34733811/155423). If not, please **[edit]** your question to explain the differences. Otherwise, we can mark this question as already answered. – Shepmaster Apr 13 '20 at 15:21
  • Thanks for pointing me in the right direction. Switching from `Iter` to `IntoIter` and calling `vec.into_iter()` helped me finish the implementation. I think the right question to ask now is if it would be possible to use `Vec` with nom combinators without intorducing an intermediate structure. Please advise me if I should make it a new question or update the current one and remove intermediate steps? – mpontus Apr 13 '20 at 16:28

0 Answers0