12

According to the Rust Reference:

The isize type is a signed integer type with the same number of bits as the platform's pointer type. The theoretical upper bound on object and array size is the maximum isize value. This ensures that isize can be used to calculate differences between pointers into an object or array and can address every byte within an object along with one byte past the end.

This obviously constrain an array to at most 2G elements on 32 bits system, however what is not clear is whether an array is also constrained to at most 2GB of memory.

In C or C++, you would be able to cast the pointers to the first and one past last elements to char* and obtain the difference of pointers from those two; effectively limiting the array to 2GB (lest it overflow intptr_t).

Is an array in 32 bits also limited to 2GB in Rust? Or not?

Nayuki
  • 17,911
  • 6
  • 53
  • 80
Matthieu M.
  • 287,565
  • 48
  • 449
  • 722

2 Answers2

7

The internals of Vec do cap the value to 4GB, both in with_capacity and grow_capacity, using

let size = capacity.checked_mul(mem::size_of::<T>())
                   .expect("capacity overflow");

which will panic if the pointer overflows.

As such, Vec-allocated slices are also capped in this way in Rust. Given that this is because of an underlying restriction in the allocation API, I would be surprised if any typical type could circumvent this. And if they did, Index on slices would be unsafe due to pointer overflow. So I hope not.

It might still not be possible to allocate all 4GB for other reasons, though. In particular, allocate won't let you allocate more than 2GB (isize::MAX bytes), so Vec is restricted to that.

Vikramaditya Gaonkar
  • 717
  • 10
  • 23
Veedrac
  • 58,273
  • 15
  • 112
  • 169
  • 1
    So the answer is: yes this is limited to 2GB? – Matthieu M. Sep 01 '15 at 16:39
  • @MatthieuM. Yes. I've edited my answer to be a bit clearer about this. – Veedrac Sep 01 '15 at 16:57
  • 1
    `capacity` is a `usize` in both methods, so the maximum capacity of a `Vec` would be 4GB. However, the allocator might fail if the requested allocation size is over 2GB... – Francis Gagné Sep 01 '15 at 23:06
  • @FrancisGagné You are indeed correct, thanks. I've updated my comment. I'm not too sure about if it's even possible to allocate >2GB, but the APIs surely won't refuse you if you manage. – Veedrac Sep 01 '15 at 23:34
4

Rust uses LLVM as compiler backend. The LLVM instruction for pointer arithmetic (GetElementPtr) takes signed integer offsets and has undefined behavior on overflow, so it is impossible to index into arrays larger than 2GB when targeting a 32-bit platform.

To avoid undefined behavior, Rust will refuse to allocate more than 2 GB in a single allocation. See Rust issue #18726 for details.

Daniel
  • 15,944
  • 2
  • 54
  • 60
  • 2
    I checked the bug and the GEP FAQ, and the limit is not a matter of memory but of number of elements: when you pass `i32 1` to GEP it steps by 1 element, so with 4 bytes elements it could address into a 4 GB array easily. Note that the example was specifically about `Vec` which is a 1 byte element, thus the 2GB limit for this example. – Matthieu M. Sep 02 '15 at 13:35