6

AFAIK V8 has a known hard limit on the length of allowed Strings. Trying to parse >500MB Strings will pop the error:

Invalid String Length

Using V8 flags to increase the heap size doesn't make any difference

$ node --max_old_space_size=5000 process-large-string.js

I know that I should be using Streams instead. However is there any way to increase the maximum allowed String length anyway?

Update: Answer from @PaulIrish below indicates they upped it to 1GB - but it's still not user-configurable

nicholaswmin
  • 21,686
  • 15
  • 91
  • 167

2 Answers2

14

In summer 2017, V8 increased the maximum size of strings from ~256MB to ~1GB. Specifically, from 2^28 - 16 to 2^30 - 25 on 64-bit platforms. V8 ticket.

This change landed in:

  • V8: 6.2.100
  • Chromium: 62.0.3167.0
  • Node.js: 9.0.0
Paul Irish
  • 47,354
  • 22
  • 98
  • 132
  • 9
    This appears to have regressed back tp 2^28 due to pointer compression and recently bumped back up to 2^29 (0.5GB) as per https://github.com/v8/v8/commit/ea56bf5513d0cbd2a35a9035c5c2996272b8b728 :( – Daniel Sokolowski Feb 06 '20 at 02:34
10

Sorry, no, there is no way to increase the maximum allowed String length.

It is hard-coded in the source, and a lot of code implicitly relies on it, so while allowing larger strings is known to be on people's wishlist, it is going to be a lot of work and won't happen in the near future.

jmrk
  • 34,271
  • 7
  • 59
  • 74
  • 1
    Do you happen to know the reason for this limitation? Why does `String` have that particular max length in V8? – nicholaswmin Jun 14 '17 at 13:18
  • 1
    It dates back all the way to 2010. Back then, 512MB was the overall heap limit, and I guess it was a no-brainer to decide that no single string could/should be bigger than the heap ;-) (512 MB = memory consumption of a UTF-16 string with length 2^28.) – jmrk Jun 14 '17 at 15:13
  • 2
    It's safe to assume you're on the V8 team. Thanks for everything :) – nicholaswmin Jun 14 '17 at 17:31
  • 2
    This is no longer the case. See my answer. – Paul Irish Dec 18 '17 at 17:25
  • @PaulIrish well , it actually still IS the case that there is a maximum limit; the limit was increased to roughly 1gig – guido Apr 27 '21 at 13:07