11

I have the next code:

struct Tokenizer {
    reader: BufReader<File>,
    buf: Vec<u8>,
    token: String
}

impl Tokenizer {
    fn new(path: &PathBuf) -> Tokenizer {
        let file = File::open(path).expect("Unable to open file!");
        Tokenizer {
            reader: BufReader::new(file),
            buf: Vec::<u8>::new(),
            token: String::new()
        }
    }

    fn next(&mut self) -> bool {
        if self.buf.len() == 0 {
            if self.reader.read_until(b'\n', &mut self.buf)
                .expect("Unable to read file") == 0 {
                return false;
            }
        }
        let s = String::from_utf8(self.buf).expect("Unable to read file");
        let mut start: i8 = -1;
        let mut end: i8 = -1;
        for (i, c) in s.char_indices() {
            if start == -1 {
                if !c.is_whitespace() {
                    start = i as i8;
                }
            } else {
                if c.is_whitespace() {
                    end = i as i8;
                }
            }
        }
        self.token = s.chars().skip(start as usize).take((end - start) as usize).collect();
        self.buf = s.into_bytes();
        self.buf.clear();
        return true;
    }
}

But it doesn't work due to error:

error[E0507]: cannot move out of `self.buf` which is behind a mutable reference
  --> src\parser.rs:28:35
   |
28 |         let s = String::from_utf8(self.buf).expect("Unable to read file");
   |                                   ^^^^^^^^ move occurs because `self.buf` has type `std::vec::Vec<u8>`, which does not implement the `Copy` trait

I've read similar questions that advice to use swap but I didn't get how it will help me. How to fix this error and why does it not compile?

Denis Sologub
  • 7,277
  • 11
  • 56
  • 123

2 Answers2

16

String::from_utf8 takes its parameter by value, meaning it does not merely use the vec it's given it consumes it.

But here it can't consume the input: since self is just a mutable borrow self.buf is at most a mutable borrow (aka &mut Vec<u8>), and thus the signatures don't match. This is because String::from_utf8 will check if the input is valid and then just reinterpret it as a valid string, avoiding new allocations.

There are two easy ways to fix this issue I can see here:

The simplest is to use a function which takes a reference as input like str::from_utf8, this will allocate a new String internally in order to return the data, therefore is a bit less efficient but more convenient.

The alternative -- and the swap suggestion you got -- is to move the data out of the borrow: if you have a mutable borrow you can't consume self.buf but you can replace it with a new (owned) buffer.

swap performs this operation and gives back the old value, so you can swap the current content of self.buf for a brand new (empty) vector, and get out the vector formerly known as self.buf, now owned by you (instead of self) and thus consumable.

The way it would look here is:

        let mut buf = Vec::new();
        swap(&mut buf, self.buf);
        // here we now consume the swapped buf
        let s = String::from_utf8(buf).expect("Unable to read file");
        let mut start: i8 = -1;
        let mut end: i8 = -1;
        for (i, c) in s.char_indices() {
            if start == -1 {
                if !c.is_whitespace() {
                    start = i as i8;
                }
            } else {
                if c.is_whitespace() {
                    end = i as i8;
                }
            }
        }
        self.token = s.chars().skip(start as usize).take((end - start) as usize).collect();
        self.buf = s.into_bytes();
        self.buf.clear();
        return true;

Addendum: since 1.40, there's also a mem::take utility function which will simply swap the value with its default (so only works for Default types).

Since Vec<u8> is Default, you could replace

        let mut buf = Vec::new();
        swap(&mut buf, self.buf);

with

        let buf = take(self.buf);
Flavio Vilante
  • 5,131
  • 1
  • 11
  • 15
Masklinn
  • 34,759
  • 3
  • 38
  • 57
  • 2
    I think you're mistaken about `str::from_utf8`. From what i can tell, it doesn't make any new allocations or create a string, it just reinterprets an existing slice of memory as a `str` slice. It is however still `O(n)` because it scans the whole slice to check that it's valid utf8. There's an unsafe version `str::from_utf8_unchecked` which is `O(1)`, which I would choose for any sizeable string. – A. Kriegman Dec 12 '21 at 17:40
2

You're getting this error as String::from_utf8 tries to take ownership of it's parameter. In your case it's property of &mut self, which is referred (you're planning to use this struct in future, not consume it).

Easy solution would be to use other function which takes reference:

fn next(&mut self) -> bool {
    ...
    let s = String::from_utf8_lossy(&self.buf[..]);
    ...
}

and remove self.buf = s.into_bytes(); line as you're 1. cleaning it on next line; 2. it was not consumed, but referenced.

Ivan Temchenko
  • 814
  • 1
  • 9
  • 12