Rust core dump when handling big files(70GB x 4)

Question

When I try to save a large (about 70GB) files in Rust, it causes a core dump and stops the process. At first I thought it might be a lack of storage, so I investigated, but it was not. Here is how the error I got.

memory allocation of 73603432908 bytes failed
/var/spool/uge/at163/job_scripts/12220153: line 8: 46483 Aborted

What I am using is

$ rustup --version
rustup 1.23.1 (3df2264a9 2020-11-30)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.50.0 (cb75ad5db 2021-02-10)`.
info: The currently active `rustc` version is `rustc 1.50.0 (cb75ad5db 2021-02-10)`.

Also, the usage crate uses ndarray. When saving the file, it is saved in .npy format using ndarray-npy.

I explain how I use ndarray-npy to store my data. The code is as follows

use ndarray; //0.14.0
use ndarray::Array3;
use ndarray_npy::write_npy; //0.7.1


struct Features {
    input_o: Array3<f32>, // 70GB
    input_c: Array3<f32>, // 70GB
    input_ca: Array3<f32>, // 70GB
    input_n: Array3<f32>,  // 70GB
    target_o: Array3<f32>, 
    target_c: Array3<f32>,
    target_ca: Array3<f32>,
    target_n: Array3<f32>,
}

fn main() {
    // do something
    write_npy(
        &(dir.to_string() + &fname + "_input_C.npy"),
        &features.input_c,
    )
    .unwrap();
    write_npy(
        &(dir.to_string() + &fname + "_input_CA.npy"),
        &features.input_ca,
    )
    .unwrap();
    write_npy(
        &(dir.to_string() + &fname + "_input_N.npy"),
        &features.input_n,
    )
    .unwrap();
    write_npy(
        &(dir.to_string() + &fname + "_input_O.npy"),
        &features.input_o,
    )
    .unwrap();

    write_npy(
        &(dir.to_string() + &fname + "_target_C.npy"),
        &features.target_c,
    )
    .unwrap();
    write_npy(
        &(dir.to_string() + &fname + "_target_CA.npy"),
        &features.target_ca,
    )
    .unwrap();
    write_npy(
        &(dir.to_string() + &fname + "_target_N.npy"),
        &features.target_n,
    )
    .unwrap();
    write_npy(
        &(dir.to_string() + &fname + "_target_O.npy"),
        &features.target_o,
    )
    .unwrap();

}

The error seems relatively clear: something tries to allocate a buffer the size of the entire thing, and that blows up because trying to commit 73GB is a bit much on most systems. You should do some debugging to track down the exact bit which fails, then either fix your code or report the issue to whatever library you're invoking (possibly, after checking if there is maybe an other API which would work instead). It seems odd that this would occur when trying to *save* the data though, I'd expect the issue to occur when loading the data in memory. — Masklinn, Mar 19 '21 at 08:14
Can you show how you're writing the data to a file? It looks like you're trying to store it all in e.g. a `Vec` before writing to disk. — sebpuetz, Mar 19 '21 at 08:55
@sebpuetz Thank you for your comment!! I updated how to save data. — musako, Mar 19 '21 at 09:04
@Masklinn Thank you for your commet! As far as I can see from the log of the code I wrote, I am receiving an error at the stage of saving the data. — musako, Mar 19 '21 at 09:06
@BoKuToTsuZenU I don't think SO will be able to help much, you'd probably get more assistance reporting this issue to the maintainer of ndarray_npy and seeing if they have suggestions for debugging or maybe lower-level methods you could use which would avoid the allocations. Running something like heap profiling ([see here](https://stackoverflow.com/questions/38254937/how-do-i-debug-a-memory-issue-in-rust) might also help. — Masklinn, Mar 19 '21 at 09:54
Also maybe trying to save to an npz? If you're going to allocate the entire size of the output, maybe a compressed version would be less of an issue. — Masklinn, Mar 19 '21 at 09:55
thank you for your comment! I will make issue in ndarray-npy — musako, Mar 19 '21 at 18:06

score 2 · Answer 1 · answered Mar 19 '21 at 09:19

2

Unless your machine has more than 280GB RAM, you won't be getting to the serialization part and the allocation error happens when trying to reserve the memory required for one of the arrays. If you are trying to work with data stored on-disk, you can check out the memmap crate to keep it on disk but treat it as if it were in memory.

answered Mar 19 '21 at 09:19

sebpuetz

2,430
1
7
15

Thank you for writing the answer. I also ran the memory request with 512GB, thinking at first that there was not enough memory and the process was failing, but I got the same result. I will try to rewrite it using memmap as you suggested. – musako Mar 19 '21 at 09:29
If you have 512GB physical RAM then it's rather strange. Maybe you have non-unlimited `ulimit`s on your RSS and / or VSS? – Masklinn Mar 19 '21 at 09:59
@Masklinn Thanks for the comment! The ulimit is a squid. It's probably not a problem. data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 131072 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) unlimited virtual memory (kbytes, -v) 4194304 file locks (-x) unlimited – musako Mar 19 '21 at 18:05

Rust core dump when handling big files(70GB x 4)

1 Answers1