How to place an array of structs at the end of another struct without memory segmentation

Question

I'm in the process of porting a low latency real-time application to Rust. Essentially I want to do as much as possible of the operations without copying. A major source of memory allocations right now are coming from the network stack. Every time a packet is received or send a lot of copying goes on under the hood to force and retrieve the bytes from a Rust structs. I want to stop doing this.

Essentially I want to be able to just convert the packets from/to bytes and transmit it. Below is a non-working example code that illustrates what I would like to do.

#[repr(C, align(4))]
#[derive(Debug)]
struct Content {
    first: u8,
    padding: [u8; 3]
}

#[repr(C, align(4))]
#[derive(Debug)]
struct Packet {
    header_id: u16,
    header_padding: [u8; 2],
    // This is another pointer I want it to be located after header_padding
    content: Box<[Content]>,
}

#[derive(Debug)]
struct ParsedPacket {
    content_length: usize,
    // The box below should directly reference the original u8 bytes without copying
    packet: Box<Packet>,
}

unsafe fn buf_to_packet(bs: Box<[u8]>) -> Option<Box<ParsedPacket>> {
    // How to implement this
    // First 4 bytes are the header
    // The rest must be 4 byte multiple and each 4 byte is a Content struct
    // So from the length of the boxed slice we can infer the content_length
    todo!()
}

unsafe fn packet_to_buf(bs: ParsedPacket) -> Box<[u8]> {
    todo!()
}

fn main() {
    let bytes = vec![
        // Header
        0, 23, 0, 0,
        // Content 0
        0, 0, 0, 0,
        // Content 1
        1, 0, 0, 0,
        // Content 2
        0, 0, 0, 0
    ].into_boxed_slice();

    let packet = unsafe { buf_to_packet(bytes) };

    println!("{:?}", content);
}

There are two problems with this. Rust requires the content to be a boxed slice. Which mains it necessarily isn't located in the struct itself but might be somewhere else. Also a boxed slice contains it's own bookkeeping about the length. I guess I need to do my own bookkeeping with the length. But I don't know how I can add a struct member with dynamic length without using a boxed slice.

I am of course willing to use unsafe to do it. But I'm not experienced with Rust so I don't know how I can actually implement this.

Essentially what I'm looking for in Rust is Flexible array members from C.

Answer 1

In Rust this is called a dynamically sized type (DST) which commonly covers trait objects and slices but you can make custom DSTs even though support spotty at best (through use of unsafe and nightly features for functionality like this).

To make your struct into a custom DST (and act similar to a flexible array member in C), you simply make the last member into a bare slice like so:

#[repr(C, align(4))]
#[derive(Debug)]
struct Packet {
    header_id: u16,
    header_padding: [u8; 2],
    content: [Content],
}

To "cast" from bytes to a Packet the bytes must first be aligned. This will commonly be 8-byte aligned already on many modern systems, but in case that can't be relied on, you can use the trick here: How do I allocate a Vec that is aligned to the size of the cache line? Because of alignment and allocation restrictions, its much easier to create a &Packet than a Box<Packet>.

Then to properly manipulate "fat pointers" that are used to express indirection to DSTs, we'll need to rely on the unstable feature ptr_metadata. Putting that to use yields this:

#![feature(ptr_metadata, pointer_is_aligned)]

#[repr(C, align(4))]
#[derive(Debug)]
struct Content {
    first: u8,
    padding: [u8; 3]
}

#[repr(C, align(4))]
#[derive(Debug)]
struct Packet {
    header_id: u16,
    header_padding: [u8; 2],
    content: [Content],
}

fn buf_to_packet(bytes: &[u8]) -> Option<&Packet> {
    const PACKET_HEADER_LEN: usize = 4;

    let bytes_len = bytes.len();
    let bytes_ptr = bytes.as_ptr();
    if bytes_len < PACKET_HEADER_LEN || !bytes_ptr.is_aligned_to(4) {
        return None;
    }

    let content_len = (bytes_len - PACKET_HEADER_LEN) / std::mem::size_of::<Content>();
    let packet_raw: *const Packet = std::ptr::from_raw_parts(bytes_ptr.cast(), content_len);

    // SAFETY: the pointer is aligned and initialized
    unsafe { packet_raw.as_ref() }
}

See also:

• Here is a code sample that demonstrates what you'd need to know if you actually needed to create a Box<Packet> from a Box<u8>.
• How to create a smart pointer to an unsized type with an embedded slice? for more info how you might allocate a Box<Packet> from nothing.