4

In 2018 a Tech Lead at Google said they were working to "support buffers way beyond 4GiB" in V8 on 64 bit systems. Did that happen?

Trying to load a large file into a buffer like:

const fileBuffer = fs.readFileSync(csvPath);

in Node v12.16.1 and getting the error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (3461193224) is greater than possible Buffer: 2147483647 bytes.

and in Node v14.12.0 (latest) and getting the error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (3461193224) is greater than 2 GB

Which looks to me to be a limit set due to 32 bit integers for addressing of the buffers. But I don't understand why this would be a limitation on 64 bit systems... Yes I realize I can use streams or read from the file at a specific address, but I have massive amounts of memory laying around, and I'm limited to 2147483647 bytes because Node is limited at 32 bit addressing?

Surely having a buffer of a high frequency random access data-set fully loaded into a buffer rather than streamed has performance benefits. The code involved in directing the request to pull from the multiple buffer alternative structure is going to cost something, regardless of how small...

I can use the --max-old-space-size=16000 flag to increase the maximum memory used by Node, but I suspect this is a hard-limit based on the architecture of V8. However I still have to ask since the tech lead at Google did claim they were increasing the maximum buffer size past 4GiB: Is there any way in 2020 to have a buffer beyond 2147483647 bytes in Node.js?

Edit, relevant tracker on the topic by Google, where apparently they were working on fixing this since at least last year: https://bugs.chromium.org/p/v8/issues/detail?id=4153

J.Todd
  • 707
  • 1
  • 12
  • 34
  • 1
    Which version of Node are you using? – tadman Sep 25 '20 at 00:51
  • @tadman v12.16.1 – J.Todd Sep 25 '20 at 00:52
  • @tadman why, does 14x support it? – J.Todd Sep 25 '20 at 00:53
  • This article might help: https://itnext.io/using-node-js-to-read-really-really-large-files-pt-1-d2057fe76b33 – paulsm4 Sep 25 '20 at 00:56
  • 1
    I tested with Node v14.12.0 (latest) and got only a slightly modified error: `RangeError [ERR_FS_FILE_TOO_LARGE]: File size (3461193224) is greater than 2 GB` – J.Todd Sep 25 '20 at 00:57
  • It was just worth checking as these things can be updated from time to time. I've hit limits in Node.js before and the only way I could work around it was to write C or C++ code that did the heavy lifting, using Node as a wrapper. – tadman Sep 25 '20 at 00:58
  • @paulsm4 I appreciate the article, the use of streams and specific location reads on a file is something I'm aware of and I still point out that those alternate methods do come with some performance hits vs just holding the entire file in a single buffer. Assuming we have the memory available, it seems we should be able to leverage our 64 bit systems to design our servers with the maximum theoretical cache efficiency. 64 bit range addressing is beyond the reach of Node on 64 bit systems in 2020? According to a tech lead (who seems to still be working at Google) this shouldn't be the case. – J.Todd Sep 25 '20 at 01:06

1 Answers1

5

Did that happen?

Yes, V8 supports very large (many gigabytes) ArrayBuffers nowadays.

Is there any way to have a buffer beyond 2147483647 bytes in Node.js?

Yes:

$ node
Welcome to Node.js v14.12.0.
Type ".help" for more information.
> let b = Buffer.alloc(3461193224)
undefined
> b.length
3461193224

That said, it appears that fs.readFileAsync has its own limit: https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L5 I have no idea what it would take to lift that. I suggest you file an issue on Node's bug tracker.

FWIW, Buffer has yet another limit:

> let buffer = require("buffer")
undefined
> buffer.kMaxLength
4294967295

And again, that's Node's decision, not V8's.

jmrk
  • 34,271
  • 7
  • 59
  • 74