Point FileReader.readAsArrayBuffer to WebAssembly's linear memory

Question

I'm working with WebAssembly right now, and wanted to be able to pass a file's bytes from JavaScript to Rust code. Currently, I've exposed the following Rust struct to JavaScript:

use js_sys::{Function, Uint8Array};
use wasm_bindgen::prelude::{wasm_bindgen, JsValue};

#[wasm_bindgen]
pub struct WasmMemBuffer {
    buffer: Vec<u8>,
}

#[wasm_bindgen]
impl WasmMemBuffer {
    #[wasm_bindgen(constructor)]
    pub fn new(byte_length: u32, f: &Function) -> Self {
        let mut buffer = vec![0; byte_length as usize];
        unsafe {
            let array = Uint8Array::view(&mut buffer);
            f.call1(&JsValue::NULL, &JsValue::from(array))
                .expect("The callback function should not throw");
        }

        Self { buffer }
    }
}

This struct, WasmMemBuffer, will point to a place in WebAssembly memory, meaning that I can write to it from JavaScript without any expensive copy operations.

import { WasmMemBuffer } from 'rust-crate';

const buf = new WasmMemBuffer(1000, (arr: Uint8Array) => {
  for (let i = 0; i < arr.length; i++) {
    arr[i] = 1.0; // Initialize the buffer with ones
  }
});

This works great. However, I'd like to read a user-submitted file's contents directly into WebAssembly's linear memory, without creating an ArrayBuffer, converting it to a Uint8Array, and copying its values to the WasmMemBuffer. As I understand it, FileReader allocates a new ArrayBuffer when calling readAsArrayBuffer. I'd like to point it to the WasmMemBuffer instead. Is there a way I can do this? For reference, this is the code I'm using to read the file.

// e is the onchange event from a <input type="file"> element
function readFile(e: Event) {
  const target = e?.target as HTMLInputElement;
  const file = target?.files?.[0];

  if (!target || !file) {
    return;
  }

  const reader = new FileReader();

  reader.onload = (e) => {
    const res = e.target?.result as ArrayBuffer;
    
    if (res) {
      // Do something with res
    }  
  };

  reader.readAsArrayBuffer(file);
}

I suspect you can do this through [`ReadableStreamBYOBReader`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamBYOBReader/read), but the details are beyond me. Alternatively, I think you can also pass the `file` object to wasm, and cast it via `gloo_file::File::from(file.dyn_into::().unwrap())`, which will allow you to call read directly from wasm (but I have no idea how it copies the bytes). Btw, [related answer](https://stackoverflow.com/questions/75141183/how-to-select-a-file-as-bytes-or-text-in-rust-wasm/75143050#75143050) with a completely different approach. — Caesar, Feb 08 '23 at 00:11
Looks `ReadableStreamBYOBReader` [isn't supported](https://caniuse.com/?search=ReadableStreamBYOBReader) on Safari. Also, I don't think it would allow me to use the existing `Uint8Array` created in Rust. — acd, Feb 08 '23 at 15:27

score 1 · Accepted Answer · answered Feb 28 '23 at 03:56

Forgot about this question, but I figured it out by using a chunked FileReader wrapper. This also has the pleasant effect of allowing very large files to be read in a much more efficient manner.

Here's the class:

import { WasmByteBuffer } from 'rs';

export default class ChunkedBuffer {
  readonly buffer: WasmByteBuffer;
  private _chunkSize: number;
  private _input: Blob;
  private _bufferArray: Uint8Array | null = null;
  private _chunk: Blob | null = null;
  private _reader: FileReader = new FileReader();
  private _offset = 0;
  private _process: () => void;

  constructor(
    input: Blob,
    process: () => void = () => undefined,
    chunkSize = 1024,
  ) {
    if (chunkSize <= 0) {
      throw new RangeError('chunkSize should be larger than zero');
    }

    this._chunkSize = chunkSize;
    this._input = input;
    this._process = process;

    this.buffer = new WasmByteBuffer(input.size, (arr: Uint8Array) =>
      this._init(arr),
    );
  }

  private _init(arr: Uint8Array) {
    this._bufferArray = arr;

    this._reader.onload = () => {
      const b = this._reader.result as ArrayBuffer;

      if (!b) {
        return;
      }

      const res = new Uint8Array(b);
      this._fill(res);

      this._offset += this._chunkSize;
      this._read();
    };

    this._read();
  }

  private _fill(res: Uint8Array) {
    if (!this._bufferArray) {
      return;
    }

    for (let i = 0; i < res.byteLength; i++) {
      this._bufferArray[this._offset + i] = res[i];
    }
  }

  private _read() {
    if (this._offset > 1e9) {
      throw new Error(`File size must not exceed 1GB`);
    }

    if (this._offset > this._input.size) {
      this._process();
      return;
    }

    this._chunk = this._input.slice(
      this._offset,
      this._offset + this._chunkSize,
    );
    this._reader.readAsArrayBuffer(this._chunk);
  }
}

Point FileReader.readAsArrayBuffer to WebAssembly's linear memory

1 Answers1