8

I know crypto.subtle.digest could be used to generate a digest of a given ArrayBuffer.

However, when the file is large, e.g. 5GB, I always get this error

Uncaught (in promise) DOMException: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.

click https://jsfiddle.net/kq4s2utf/ to see the full version.

How do I solve this?

PutBere
  • 131
  • 1
  • 12
  • 1
    its for hours i am working on it. but i couldnt fix the *MEMORY* error. but i will – Abilogos Dec 07 '20 at 17:40
  • 1
    It seems like there was a question like this before: https://stackoverflow.com/questions/52738019/hash-large-files-with-crypto-subtle-digestsha-256-buffer I think the solution is to divide the file into parts and concatenate the digests. – Ethicist Dec 10 '20 at 21:28

2 Answers2

4

I believe the right answer would be to stream the file content instead of reading whole file in memory at once. Blob allows to read a file as a stream: https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream

Now the problem is that Web Cryptography API you're using doesn't support streams or incremental hashing. There is a long (and quite old) discussion about that with no clear outcome: https://github.com/w3c/webcrypto/issues/73 .

I would suggest using some 3rd party library that supports incremental hashing. E.g. https://github.com/Caligatio/jsSHA

The resulting code could look like

async function calcDigest() {
    const reader = finput.files[0].stream().getReader()
    const shaObj = new jsSHA("SHA-256", "ARRAYBUFFER")

    while (true) {
        const {done, value} = await reader.read()
        if (done) break
        shaObj.update(value)
    }

    console.log(shaObj.getHash("HEX"))
}
amakhrov
  • 3,820
  • 1
  • 13
  • 12
0

One possible solution could be compressing the ArrayBuffer through LZMA (LZMA-JS) or something and creating a digest of that instead of the complete data buffer. I haven't had the time to test this out because I didn't have any files larger than 4GB, but it could work.

Ethicist
  • 791
  • 2
  • 7
  • 23