2

I'm trying to create a container object containing a const array using a const initializer function for each element.

For arrays of a fixed size (notated with an integer literal) this is already solved; the twist here is that the length of the array is a const generic.

A further twist is that I need this entire thing in no_std (core) mode.

Here is a code example that demonstrates my problem:

// Important: Element is **not** `Copy`.
#[derive(Debug)]
struct Element(usize);
impl Element {
    pub const fn array_initializer(pos: usize) -> Self {
        Element(pos)
    }
}

#[derive(Debug)]
pub struct Container<const N: usize> {
    data: [Element; N],
}

impl<const N: usize> Container<N> {
    pub const fn new() -> Self {
        // The content of this function is the only
        // part that is open for change!

        // Task is: create a container with `N` elements,
        // where every element is initialized with
        // `Element::array_initializer`.
    }
}

static STATIC_CONTAINER: Container<5> = Container::new();

fn main() {
    println!("{:?}", STATIC_CONTAINER);
}

Desired output:

Container { data: [Element(0), Element(1), Element(2), Element(3), Element(4)] }

Of course there also might be the possibility that this is entirely impossible in Rust's current state; although I would be quite sad about it.

I don't care about the use of unsafe in the Container::new function, as long as it is sound.


Things I've tried so far:

  • array-const-fn-init

    It fails because it does not accept const generic array lengths.

    impl<const N: usize> Container<N> {
        pub const fn new() -> Self {
            const fn element_init(pos: usize) -> Element {
                Element::array_initializer(pos)
            }
    
            let data: [Element; N] = array_const_fn_init::array_const_fn_init!(element_init; N);
    
            Self { data }
        }
    }
    
    error: proc macro panicked
      --> src/main.rs:21:34
       |
    21 |         let data: [Element; N] = array_const_fn_init::array_const_fn_init!(element_init; N);
       |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       |
       = help: message: Expected <usize>, found N
    
  • MaybeUninit, based on this example

    impl<const N: usize> Container<N> {
        pub const fn new() -> Self {
            use core::mem::MaybeUninit;
    
            let mut data: [MaybeUninit<Element>; N] = unsafe { MaybeUninit::uninit().assume_init() };
    
            {
                // const for's are not stabilized yet, so use a loop
                let mut i = 0;
                while i < N {
                    data[i] = MaybeUninit::new(Element::array_initializer(i));
                    i += 1;
                }
            }
    
            let data: [Element; N] =
                unsafe { core::mem::transmute::<[MaybeUninit<Element>; N], [Element; N]>(data) };
    
            Self { data }
        }
    }
    
    error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
      --> src/main.rs:30:22
       |
    37 |             unsafe { core::mem::transmute::<[MaybeUninit<Element>; N], [Element; N]>(data) };
       |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       |
       = note: source type: `[MaybeUninit<Element>; N]` (this type does not have a fixed size)
       = note: target type: `[Element; N]` (this type does not have a fixed size)
    

    This is technically nonsense and probably a compiler restriction that will be lifted in future; [MaybeUninit<Element>; N] and [Element; N] definitely do have the same size.

  • core::intrinsics::transmute_unchecked

    As we know the size is identical, use transmute_unchecked to do the transmutation anyway.

    Now we are in nightly and feature territory, which sadly isn't compatible with my situation :/

    #![feature(core_intrinsics)]
    
    // ...
    
    impl<const N: usize> Container<N> {
        pub const fn new() -> Self {
            use core::mem::MaybeUninit;
    
            let mut data: [MaybeUninit<Element>; N] = unsafe { MaybeUninit::uninit().assume_init() };
    
            {
                // const for's are not stabilized yet, so use a loop
                let mut i = 0;
                while i < N {
                    data[i] = MaybeUninit::new(Element::array_initializer(i));
                    i += 1;
                }
            }
    
            let data: [Element; N] = unsafe {
                core::intrinsics::transmute_unchecked::<[MaybeUninit<Element>; N], [Element; N]>(data)
            };
    
            Self { data }
        }
    }
    

    This does give us the correct result, but feature(core_intrinsics) probably won't ever get included in stable.

  • MaybeUninit::array_assume_init

    This does seem a little bit saner and a little bit closer to stabilization, but currently it still requires a nightly compiler and feature gates.

    #![feature(maybe_uninit_array_assume_init)]
    #![feature(const_maybe_uninit_array_assume_init)]
    
    // ...
    
    impl<const N: usize> Container<N> {
        pub const fn new() -> Self {
            use core::mem::MaybeUninit;
    
            let mut data: [MaybeUninit<Element>; N] = unsafe { MaybeUninit::uninit().assume_init() };
    
            {
                // const for's are not stabilized yet, so use a loop
                let mut i = 0;
                while i < N {
                    data[i] = MaybeUninit::new(Element::array_initializer(i));
                    i += 1;
                }
            }
    
            let data: [Element; N] = unsafe { MaybeUninit::array_assume_init(data) };
    
            Self { data }
        }
    }
    
Finomnis
  • 18,094
  • 1
  • 20
  • 27
  • You can read from a raw pointer to get around the transmute issue in the `MaybeUninit` version: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=81205f7aba22b84120e54d93a788c537 – eggyal Aug 28 '23 at 10:12
  • That's great! You wanna make this an answer? – Finomnis Aug 28 '23 at 10:23
  • Or better, use `transmute_copy()`. – Chayim Friedman Aug 28 '23 at 10:27
  • @eggyal It seems like I oversimplified my [mre]; In my real code, it throws this very weird error: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=b4bdf2b3a42ad99196e37f81af21f304 – Finomnis Aug 28 '23 at 10:36
  • @ChayimFriedman `transmute_copy` does not seem to be `const` – Finomnis Aug 28 '23 at 10:37
  • Oh, forgot it :) sorry. – Chayim Friedman Aug 28 '23 at 11:54
  • 2
    In that case, you can perform the cast via a `union`: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=0654e2192537dfa078630a77087cc4ca – eggyal Aug 28 '23 at 12:01
  • @eggyal You need to mark the union `#[repr(C)]`. – Chayim Friedman Aug 28 '23 at 12:03
  • @ChayimFriedman: Is that the case even when the union's fields are guaranteed to have the same ABI? – eggyal Aug 28 '23 at 12:55
  • 1
    @Finomnis The code linked in your comment above is unsound. You need to `forget()` data after the type conversion – otherwise everything in the array will be dropped in place. It also may leak resources if `array_initializer()` panics. I think you want to use the nightly compiler to implement this, which would turn this problem from really tricky to mostly straightforward. – Sven Marnach Aug 28 '23 at 13:02
  • I'd personally just not do that in const code yet, and instead run a lazy initializer. – Sven Marnach Aug 28 '23 at 13:03
  • Here's what I think is a sound implementation for Nightly: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=f59e6c32e84c5beed75f8d59a66557ad Maybe using `std::array::from_fn()` is the more attractive option after all. – Sven Marnach Aug 28 '23 at 13:43
  • `std::array::from_fn()` isn't `const`, though ... I agree that this would be the cleanest solution – Finomnis Aug 28 '23 at 19:51
  • @eggyal Yes! The `union` trick absolutely did it. Thank you! – Finomnis Aug 28 '23 at 20:15
  • @eggyal Yes, even then. The layout of `#[repr(Rust)]` unions is not guaranteed. – Chayim Friedman Aug 28 '23 at 21:54
  • @SvenMarnach `MaybeUninit` doesn't drop the data, so this is not unsound. The point about leaking still stays, though. – Chayim Friedman Aug 28 '23 at 21:56
  • @ChayimFriedman I think he meant my pointer cast based conversion – Finomnis Aug 29 '23 at 05:10

1 Answers1

0

The ideal solution would be either:

So at the current time, it seems that a union based transmutation is the most sane option I could find:

// Important: Element is **not** `Copy`.
#[derive(Debug)]
struct Element(usize);
impl Element {
    pub const fn array_initializer(pos: usize) -> Self {
        Element(pos)
    }
}

#[derive(Debug)]
pub struct Container<const N: usize> {
    data: [Element; N],
}
impl<const N: usize> Container<N> {
    pub const fn new() -> Self {
        use core::mem::{ManuallyDrop, MaybeUninit};

        let mut data: [MaybeUninit<Element>; N] = unsafe { MaybeUninit::uninit().assume_init() };

        {
            // const for's are not stabilized yet, so use a loop
            let mut i = 0;
            while i < N {
                data[i] = MaybeUninit::new(Element::array_initializer(i));
                i += 1;
            }
        }

        #[repr(C)]
        union InitializedData<const N: usize> {
            uninit: ManuallyDrop<[MaybeUninit<Element>; N]>,
            init: ManuallyDrop<[Element; N]>,
        }
        let data = ManuallyDrop::into_inner(unsafe {
            InitializedData {
                uninit: ManuallyDrop::new(data),
            }
            .init
        });

        Self { data }
    }
}

static STATIC_CONTAINER: Container<5> = Container::new();

fn main() {
    println!("{:?}", STATIC_CONTAINER);
}
Container { data: [Element(0), Element(1), Element(2), Element(3), Element(4)] }

It is important here that Element::array_initializer can't panic, otherwise there might have to be additional considerations regarding Drop behaviour.

Finomnis
  • 18,094
  • 1
  • 20
  • 27
  • Honest question – does this really need to run at compile time, as opposed to a lazily initialized static array? I'd personally need a very strong reason to put this code into production if I could replace it by a one-liner that runs once (and might even reduce the binary size, since you don't need to embed the data). – Sven Marnach Aug 29 '23 at 06:48
  • @SvenMarnach How do lazy init something in a `no_std` environment that doesn't provide locks or `alloc`? There are situations where having something like this is a very useful tool to have. If you want more information about the specific usecase, [here you go](https://github.com/crossbeam-rs/crossbeam/pull/1024). In this specific case, binary size is even useful because it means that it will put the data structure into global static memory opposed to the stack. Having a static place for memory is a good thing in embedded. – Finomnis Aug 30 '23 at 06:04
  • To be precise: in bare-metal embedded that has no OS running behind it. – Finomnis Aug 30 '23 at 06:09
  • There are basically two cases: Either, you need concurrency in your envionment, in which case you need locks anyway, regardless of what else you are doing. Or you don't need concurrency, in which case you can simply use `core::cell::OnceCell`. (And for what it's worth, `critical_section::Mutex` works virtually everywhere.) I still struggle to imagine a case where I would actually go for the const initialization. – Sven Marnach Aug 30 '23 at 11:42
  • (Or maybe I should replace "locks" with "synchronization primitives" in my previous comment. You can do lock-free concurrency, but you can also do lock-free lazy initialization, if you are so inclined.) – Sven Marnach Aug 30 '23 at 11:52
  • @SvenMarnach In a case where you use RTIC, which has its own locks, and `critical_section` messes with its priorities. I try to debug USB actions, and using `imxrt-log` is too slow due to its internal locking. USB looses its connection, and I strongly suspect the `imxrt-log` locks to be responsible for a priority inversion (because they don't dynamically increase priority, as the RTIC locks do). So I wanted to collect messages in a lightweight lockless queue. But I don't have an `alloc` on my target, so a lock-free const initialized array based queue is exactly the tool I need. – Finomnis Aug 30 '23 at 11:53
  • But I honestly didn't come here to get my motives questioned, I just wanted to know if such a thing is possible with current Rust. I'll ask codereview if I want my design decisions checked. Yes, I accept that you suspect an XY problem here, but I think my question is a valid question. – Finomnis Aug 30 '23 at 11:55
  • 1
    Fair enough. My goal was to _understand_ your motives, rather than questioning them. Often, when I'd do something differently than other people, there is something I missed, so I keep asking until I understand. This isn't the first time this comes across the wrong way, but I keep doing it anyway, since I keep learning from these conversations. (For what it's worth, I still don't understand your motives here, but I've taken up enough of your time already.) – Sven Marnach Aug 30 '23 at 12:02
  • @SvenMarnach It's hard to explain without very much in-depth knowledge. I'm not sure how familiar you are with RTIC - the main problem is that RTIC doesn't want you to use other locks. It provides its own locks and comes with deadlock-freedom guarantees which break when you use your own locks which do not contain a priority inheritance mechanism. (That's at least my understanding). And yes, there's probably some way to dynamically allocate it, but I didn't see an advantage over compile time initialization. – Finomnis Aug 30 '23 at 12:06
  • @SvenMarnach Especially because I have to allocate the memory statically anyway, because, as I already said, I don't have `alloc`. So no heap. Only stack and static memory, and stack is potentially too small for the queue. And it's much harder to always pass references around. – Finomnis Aug 30 '23 at 12:07
  • I've used RTIC a few times on an RP2040, but I'm not too familiar with it, and I found the documentation a bit hard to navigate. Anyway, the difference we are talking about is whether to initialize the array in the init task or at compile time, neither of which should require locks at all. – Sven Marnach Aug 30 '23 at 12:16
  • I think this discussion is going too far off-topic. – Finomnis Aug 30 '23 at 14:52