4

I am working on a web assembly application in rust. The program relies on streaming a file and storing the data in memory. However I run into a rust_oom issue.

This issue only arises when recompiling the std library with atomic, bulk-memory, and mutable-global flags. Reproducable via .cargo/config.toml:

[target.wasm32-unknown-unknown]
rustflags = ["-C", "target-feature=+atomics,+bulk-memory,+mutable-globals"]

[unstable]
build-std = ["panic_abort", "std"]

Compiling without these flags works fine.

The relevant rust code:

#[wasm_bindgen]
pub async fn start(r: JsValue) {
    // javascript response into websys response
    let resp: Response = r.dyn_into().unwrap();


    // from example here https://github.com/MattiasBuelens/wasm-streams/blob/master/examples/fetch_as_stream.rs
    let raw_body = resp.body().unwrap();
    let body = ReadableStream::from_raw(raw_body.dyn_into().unwrap());
    let mut stream = body.into_stream();

    // store the data in memory

    let mut v = vec![];

    log("start streaming");

    while let Some(Ok(chunk)) = stream.next().await {
        //convert to vec<u8>
        let mut x = chunk.dyn_ref::<Uint8Array>().unwrap().to_vec();
        v.append(&mut x);
    }
    log(&format!("{}", v.len()));

    log("done streaming");
}

The full error message provided

    at rust_oom (wasm_memtest_bg.wasm:0xfa10)
    at __rg_oom (wasm_memtest_bg.wasm:0xfdd4)
    at __rust_alloc_error_handler (wasm_memtest_bg.wasm:0xfd38)
    at alloc::alloc::handle_alloc_error::rt_error::hf991f317b52eeff2 (wasm_memtest_bg.wasm:0xfdb3)
    at core::ops::function::FnOnce::call_once::ha90352dededaa31f (wasm_memtest_bg.wasm:0xfdc9)
    at core::intrinsics::const_eval_select::h04f9b6091fe1f42f (wasm_memtest_bg.wasm:0xfdbe)
    at alloc::alloc::handle_alloc_error::h82c7beb21e18f5f3 (wasm_memtest_bg.wasm:0xfda8)
    at alloc::raw_vec::RawVec<T,A>::reserve::do_reserve_and_handle::h65538cba507bb5c9 (wasm_memtest_bg.wasm:0xc149)
    at <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he4014043296c78b9 (wasm_memtest_bg.wasm:0x1c85)
    at wasm_bindgen_futures::task::multithread::Task::run::h77f1075b3537ddf9 (wasm_memtest_bg.wasm:0x5a0e)

Here is the full project if you want to test it out.

https://github.com/KivalM/memory-wasm

EDIT: The code required to reproduce the issue is provided in the first snippet. The full wasm application is provided in the repository so that you do not have to create one from scratch. I will include the rust code in the question.

The issue arises when loading a simple 500MB file into memory. Which should be possible via this approach. Yet Chrome(and chromium-based browsers) tend to run out of memory, whereas when the rustflags are not present the program runs just fine.

I will move the links here as they are not part of the question, but can be relevant.

https://rustwasm.github.io/2018/10/24/multithreading-rust-and-wasm.html https://rustwasm.github.io/wasm-bindgen/examples/raytrace.html

Kival M
  • 182
  • 1
  • 10
  • It seems that you're loading the whole file into memory. It fails for me too, and I have the issue with huge (2GB) files anyways, whether `rustflags` is there or not. You put the file into `let mut v = vec![];` in memory. Is there any chance you could stream it further without saving into memory. Thank you for great minimal reproducible example! – Igor Loskutov May 19 '22 at 10:46
  • The problematic code and an exposition of the problem needs to be included in the question itself, not behind a third-party link which could stop working or redirect to a malware site at any time. – tripleee May 19 '22 at 10:47
  • @tripleee, the problematic code is included in the first snippet. My question is why oom occurs when the rustflags are present. The code provided in the repo is just so you dont have to write the code to read a file yourself. – Kival M May 19 '22 at 11:14
  • @IgorLoskutov Thanks for the response. My intention with this code is to stream the file and load it into a dataframe. Storing the data in the vector is just meant to simulate the memory usage. Yet reading a file with these rustflags runs out of memory, whereas without them the file would read just fine. The file is fairly small at 500mb. – Kival M May 19 '22 at 11:23

0 Answers0