Allocating a buffer on the heap at runtime

Question

I am learning Rust by writing simple binary decoder.

I'm using a BufferedReader with the byteorder crate to read numbers, but I'm having problems with reading byte buffers.

I want to read byte data into buffer allocated at runtime. Then I want to pass ownership of this buffer to a struct. When struct is no longer in use, the buffer should be deallocated.

There seems to be no way to allocate array with size determined at runtime on heap except some Vec::with_capacity() hacks. Any ideas how to implement this with proper Rust semantics?

Your question seems to be closer to "How can I use the functionality of `box` in stable code". Perhaps you should update your title. — Shepmaster, Sep 05 '15 at 15:17
Just a point of "semantics": `box` *is* proper Rust, it's just not stable Rust for now :) — Matthieu M., Sep 05 '15 at 16:11

David · Answer 1 · 2020-06-09T14:21:18.507

This will create a pre-allocated mutable 500MB byte buffer of zeros stored on the heap with no need for unsafe rust:

// Correct

let mut buffer = vec![0_u8; 536870912];

Note that the following code below is not a good idea and will most likely result in a stack overflow because the buffer is created on the stack before being boxed and moved to the heap.

// Incorrect - stack used

let mut bytes: Box<[u8]> = Box::new([0_u8; 536870912])

// Incorrect - slow

let mut bytes = Vec::with_capacity(536870912);
for _ in 0..bytes.capacity() {
    bytes.push(0_u8);
}

score 6 · Accepted Answer · edited Jun 09 '20 at 14:24

6

Rust is a low-level language; thus you can allocate raw memory and then fill it with objects yourself. Of course, it will require unsafe code, as all fiddling with raw memory does.

Here is a complete example:

use std::{
    alloc::{self, Layout},
    mem, ptr,
};

fn main() {
    unsafe {
        let layout = Layout::from_size_align(512 * 1024, 4 * 1024).expect("Invalid layout");
        let mut raw: *mut i32 = mem::transmute(alloc::alloc(layout));

        for i in 0..(512 * 1024 / 4) {
            ptr::write(raw, i as i32);
            raw = raw.offset(1)
        }
    }
}

Of course, in real code, I would just use Vec to safely manage the memory for me. It's just simpler!

edited Jun 09 '20 at 14:24

Shepmaster

388,571
95
1,107
1,366

answered Sep 05 '15 at 16:20

Matthieu M.

287,565
48
449
722

1

*fill it with objects yourself* — is it worth mentioning zero-sized types and destructors and all the other complexities that explain why `Vec` is better to use? Any maybe use `as *mut i32` instead of `transmute`? And maybe use `mem::size_of` instead of hard coding `4`? And for my own curiosity, why align the memory to a 4K chunk? +1 for "just use `Vec`" ^_^. – Shepmaster Sep 05 '15 at 17:49
@Shepmaster: I hesitated in enumerating plenty of potential issues, but I was afraid it would just sound like rambling... – Matthieu M. Sep 05 '15 at 18:02
One day, you can be a professional rambler, such as myself! ;-) – Shepmaster Sep 05 '15 at 21:47

Shepmaster · Answer 3 · 2021-07-14T13:37:59.880

0

I tried using box but it seems that it is experimental and I can't use it with release branch. Any ideas how to implement this with proper Rust semantics?

This is covered in The Rust Programming Language, specifically the section "Using Box<T> to Point to Data on the Heap".

Use Box::new:

fn main() {
    let answer: Box<u8> = Box::new(42);
}

See also:

edited Jul 14 '21 at 13:37

answered Sep 05 '15 at 15:18

Shepmaster

388,571
95
1,107
1,366

Yes, `Box::new` creates variable on heap. But to my knowledge, calling `Box::new` creates variable on the stack, and function call will copy it to heap. `box ` syntax should create right on heap AFAIK. That might be a problem since I want to create 512 KB buffers. – semtexzv Sep 05 '15 at 15:34
@semtexzv can you point to some documentation or disassembly that corroborates that? The optimizers underlying Rust (provided by LLVM) are pretty strong. – Shepmaster Sep 05 '15 at 15:37
`let mut xx = Box::new([0u8;5000000]);` causes stack overflow. It should allocate 5 megs of data. But I'm probably understanding this wrong. – semtexzv Sep 05 '15 at 15:40
@semtexzv not if you [compile in release mode](http://is.gd/uHPkP0), which enables optimizations. – Shepmaster Sep 05 '15 at 15:43
@semtexzv Also, if you were aware of `Box::new` but didn't want to use it, you should have included those restrictions in your question. – Shepmaster Sep 05 '15 at 15:49
2

Problem is not using `Box::new` problem is allocating an array at runtime directly on heap. – semtexzv Sep 05 '15 at 15:53

Allocating a buffer on the heap at runtime

3 Answers3