Rust type mismatch but only if I don't use a type annotation

Question

Motivation

I want to read a stream of values for multiple files on disc. These might be CSV files, or tab-separated, or some proprietary binary format. Therefore I want my function that handles reading multiple files to take the Path -> Iterator<Data> function as an argument. If I understand correctly, in Rust I need to box the iterator, and the function itself, since they're unsized. Therefore my reading function should be (I'm just using i32 as a simple proxy for my data here):

fn foo(read_from_file: Box<dyn Fn(&Path) -> Box<dyn Iterator<Item=i32>>>) {
    panic!("Not implemented");
}

For testing, I'd rather not be reading actual files from disc. I'd like my test data to be right there in the test module. Here's roughly what I want, but I've just put it into the main of a bin project for simplicity:

use std::path::Path;

fn foo(read_from_file: Box<dyn Fn(&Path) -> Box<dyn Iterator<Item=i32>>>) {
    panic!("Not implemented");
}

fn main() {

    let read_from_file = Box::new(|path: &Path| Box::new(match path.as_os_str().to_str().unwrap() {
        "/my_files/data.csv" => vec![1, 2, 3],
        "/my_files/data_2.csv" => vec![4, 5, 6],
        _ => panic!("Invalid filename"),
    }.into_iter()));

    foo(read_from_file);
}

The error

This gives me a compilation error:

   Compiling iter v0.1.0 (/home/harry/coding/rust_sandbox/iter)
error[E0271]: type mismatch resolving `for<'r> <[closure@src/main.rs:9:35: 13:19] as FnOnce<(&'r Path,)>>::Output == Box<(dyn Iterator<Item = i32> + 'static)>`
  --> src/main.rs:15:9
   |
15 |     foo(read_from_file);
   |         ^^^^^^^^^^^^^^ expected trait object `dyn Iterator`, found struct `std::vec::IntoIter`
   |
   = note: expected struct `Box<(dyn Iterator<Item = i32> + 'static)>`
              found struct `Box<std::vec::IntoIter<{integer}>>`
   = note: required for the cast to the object type `dyn for<'r> Fn(&'r Path) -> Box<(dyn Iterator<Item = i32> + 'static)>`

For more information about this error, try `rustc --explain E0271`.
error: could not compile `iter` due to previous error

I don't really understand this. Doesn't std::vec::IntoIter implement Iterator, in which case I don't see why this is a type error?

The fix, which I also don't understand

If I add an explicit type annotation Box<dyn Fn(&Path) -> Box<dyn Iterator<Item=i32>>>, this compiles:

use std::path::Path;

fn foo(read_from_file: Box<dyn Fn(&Path) -> Box<dyn Iterator<Item=i32>>>) {
    panic!("Not implemented");
}

fn main() {

    let read_from_file : Box<dyn Fn(&Path) -> Box<dyn Iterator<Item=i32>>>
        = Box::new(|path: &Path| Box::new(match path.as_os_str().to_str().unwrap() {
        "/my_files/data.csv" => vec![1, 2, 3],
        "/my_files/data_2.csv" => vec![4, 5, 6],
        _ => panic!("Invalid filename"),
    }.into_iter()));

    foo(read_from_file);

I'm very confused by why this works. My understanding of Rust is that, in a let definition, the explicit type is optional - unless the compiler cannot infer it, in which case the compiler should emit error[E0283]: type annotations required.

Answer 1

Pointers to dynamically sized types (DSTs) like Box<dyn Iterator<Item=i32>> are "fat". A Box<std::vec::IntoIter<i32>> is not a pointer to a DST (as the size of IntoIter is known), and hence can be a "thin" pointer simply pointing to the instance of IntoIter on the heap.

The creation and usage of a fat pointer is more expensive than that of a thin pointer. This is why, as @Aplet123 mentioned, you need to explicitly tell the compiler somehow (via type annotations or an as cast) that you want to cast the thin Box<std::vec::IntoIter<i32>> pointer generated by your closure to a fat Box<dyn Iterator<Item=i32>> pointer.

Note that if you remove the let binding and create the closure in the argument list of the foo function call, then the compiler makes the closure must return a fat pointer because of the argument type expected by foo.

Answer 2

To me, this reads like a failure of type inference, since the closure is unable to infer that it needs to return a pointer to a v-table (from dyn Iterator).

However, I'd suggest that Box<dyn Foo> might not be necessary here. It's true that, since Iterator is a trait, you can't know the size of it at compile-time, in a sense, you can.

Rust "monomorphizes" generic code, which means it generates copies of generic functions/structs/etc for each concrete type it is used with. For example, if you have:

struct Foo<T> {
  value: T
}

fn main() {
  let _ = Foo { value: "hello" };
  let _ = Foo { value: 123 };
}

It's going to generate a Foo_str_'static and a Foo_i32 (roughly speaking) and substitute those in as needed.

You can exploit this to use static dispatch with generics while using traits. Your function can be rewritten as:

fn foo<F, I>(read_from_file: F)
where
  F: Fn(&Path) -> I,
  I: Iterator<Item = i32>,
{
  unimplemented!()
}

fn main() {
  // note the lack of boxing
  let read_from_file = |path: &Path| {
    // ...
  };

  foo(read_from_file);
}

This code is (very probably but I haven't benchmarked) faster, and more idiomatic, as well as making the compiler error go away.