46

I got a fatal error reading a file that was too big to fit in a buffer.

FATAL ERROR: v8::Object::SetIndexedPropertiesToExternalArrayData() length exceeds max acceptable value

Or,

RangeError: "size" argument must not be larger than 2147483647 at Function.Buffer.allocUnsafe (buffer.js:209:3)

If I try to allocate a 1GB Buffer I get the same fatal Error,

var oneGigInBytes = 1073741824;
var my1GBuffer = new Buffer(oneGigInBytes); //Crash

What is the maximum size of a Node.js Buffer class instance?

Yilmaz
  • 35,338
  • 10
  • 157
  • 202
Joel
  • 2,928
  • 2
  • 24
  • 34
  • I think in cases of large files it's reusing the same buffer instance. So it has some fixed size that keeps getting filled with data as you read it. That's what a buffer is. – Justin Thomas Jan 23 '12 at 16:20
  • Now I'm curious what the workaround is. – jcolebrand Jan 23 '12 at 16:24
  • Do you need 1GB of data in memory all at once? Why isn't streaming the file like you currently are good enough? – fent Jan 23 '12 at 22:59
  • 2
    @DeaDEnD Streaming the file may be enough. I'd like to send the file to web service that doesn't support chunk transfer encoding. It may be enough to stat() the file, find the size and set the Content-Length header, and stream the file. I was just curious why/if there was a seemingly low 1GB limit when the advantage of allocating outside the Javascript heap was that heap could only allocate at most 1GB. – Joel Jan 24 '12 at 00:57
  • 1
    I don't think any web service would not support streaming file uploads specially with sizes like 1GB. Even 10MB would not scale well at all. – fent Jan 24 '12 at 03:05
  • I'm using Amazon S3 which allows 5GB in a single PUT request. It is recommended to break files up into multipart uploads for files greater then 100MB. http://aws.amazon.com/s3/faqs/#How_much_data_can_I_store – Joel Jan 24 '12 at 12:47
  • 1
    It appears this has been raised to 2 GiB, I just tested nodejs on Linux 64 bit. – CMCDragonkai Jul 24 '17 at 05:03
  • Take a look at [my post for an update](https://stackoverflow.com/a/54857532/124486) – Evan Carroll Feb 24 '19 at 23:21

5 Answers5

36

Maximum length of a typed array in V8 is currently set to kSmiMaxValue which depending on the platform is either:

  • 1Gb - 1byte on 32-bit
  • 2Gb - 1byte on 64-bit

Relevant constant in the code is v8::internal::JSTypedArray::kMaxLength (source).

V8 team is working on increasing this even further on 64-bit platforms, where currently ArrayBuffer objects can be up to Number.MAX_SAFE_INTEGER large (2**53 - 1). See bug 4153.

Vyacheslav Egorov
  • 10,302
  • 2
  • 43
  • 45
  • actually I think this constant can be replaced without any harm with Smi::kMaxValue then on x64 you will get at least 2gb buffers. (but getting bigger buffers requires significant changes in V8). – Vyacheslav Egorov Jan 23 '12 at 16:24
  • Thanks for the link to Github. It confirms my what my testing showed. – Joel Jan 24 '12 at 16:19
  • 2
    Due to changes in V8, this is no longer the case. I'm not 100% sure exactly where it's defined currently, but I'm getting a max size of 2gb-1byte. – Alexander O'Mara Jan 16 '16 at 19:45
14

This is now documented as part of Node's buffer api, the maximum size is buffer.constants.MAX_LENGTH.

buffer.constants.MAX_LENGTH <integer> The largest size allowed for a single Buffer instance.

  • On 32-bit architectures, this value is (2^30)-1 (~1GB).
  • On 64-bit architectures, this value is (2^31)-1 (~2GB).

This value is also available as buffer.kMaxLength.

So you can figure out how big it is by doing

> (require('buffer').constants.MAX_LENGTH + 1) / 2**30
2
Evan Carroll
  • 78,363
  • 46
  • 261
  • 468
1

Seems like the current max buffer size is 2147483647 bytes aka 2.147GB

Source: https://stackoverflow.com/a/44994896/3973137 (and my own code)

Anthony
  • 13,434
  • 14
  • 60
  • 80
0

The actual maximum size of a buffer changes across platforms and versions of Node.js. You can find out what's the limit in bytes in a given platform, run this:

import buffer from "buffer";
console.log(buffer.constants.MAX_LENGTH);
KeyKi
  • 2,693
  • 3
  • 12
  • 24
Yilmaz
  • 35,338
  • 10
  • 157
  • 202
0

for me in Node v16.14.2 the maximum buffer size is 4294967296 bytes. This means that for latest versions maximum buffer size is increasing.

warun sharma
  • 1
  • 1