Can I write an Iterator that mutates itself and then yields a reference into itself?

Question

I ran into a problem that simplifies into the following:

struct MyIter {
    vec: Vec<i8>,
}

fn fill_with_useful_data(v: &mut Vec<i8>) {
    /* ... */
}

impl<'a> Iterator for MyIter {
    type Item = &'a [i8];

    fn next(&mut self) -> Option<&'a [i8]> {
        fill_with_useful_data(&mut self.vec);

        Some(&self.vec)
    }
}

fn main() {
    for slice in (MyIter { vec: Vec::new() }) {
        println!("{}", slice);
    }
}

This generates the error:

error[E0207]: the lifetime parameter `'a` is not constrained by the impl trait, self type, or predicates
 --> src/main.rs:9:6
  |
9 | impl<'a> Iterator for MyIter {
  |      ^^ unconstrained lifetime parameter

The idea is that the iterator does a bunch of work that reflects in its fields and at each step, it yields a reference into itself to the calling code. In this case I could model it as yielding a copy of the state instead of the reference, but let's pretend that's not possible or just inconveniently expensive.

Intuitively this shouldn't be a problem because the borrow checker can ensure that .next() isn't called again while the yielded reference can still be used to inspect the iterator's state, but the Iterator trait doesn't seem to provide for that sort of thing directly. Even with some permutations like only holding on to a reference to the vector in the iterator itself or making the iterator a reference or something to get the lifetimes baked into the type earlier on, I can't get anything past the borrow checker.

I read the "Iterators yielding mutable references" blogpost but I'm not sure if/how it applies to my problem that doesn't involve mutable references.

Answer 1

This is not possible. If it were allowed one could call next again and thus modify data that is also visible via & or even invalidate the reference entirely. This is because there is no connection between the self object itself and the returned reference: there is no explicit lifetime linking them.

For the compiler to reason about this and allow returning a reference into self next needs a signature like

fn next(&'a mut self) -> Option<&'a [i8]>

However, this differs from the signature of the trait which is not allowed as generic code that just takes an T: Iterator<...> cannot tell that there are different requirements on the use of the return value for some T; all have to be handled identically.

The Iterator trait is designed for return values that are independent of the iterator object, which is necessary for iterator adaptors like .collect to be correct and safe. This is more restrictive than necessary for many uses (e.g. a transient use inside a for loop) but it is just how it is at the moment. I don't think we have the tools for generalising this trait/the for loop properly now (specifically, I think we need associated types with higher rank lifetimes), but maybe in the future.