Why do i receive `Error: Cannot create a string longer than 0x3fffffe7 characters`

Question

Executing the code below throws Error: Cannot create a string longer than 0x3fffffe7 characters. My file contains only 2 arrays and composed, the whole file contains 1,3GB. I wanted to combine them by id by using something like the code below to map through the arrayMain and return each object combined with the object from arrayItems with the same id and then restructure that object to throw out the id. But I think I am bumping up against a system limit. I am fairly new to working with large data files so any help is appreciated.

const composed = arrayMain.map((d) => {
  return {
    ...d,
    data: arrayItems.filter(({ ID }) => d.ID === ID).map(({notNeeded, ...needed}) => needed),
  };
});

If anyone is wondering how my data is structured

const arrayMain = [
  {
    ID: 30574062,
    number: 28234702,
    place: London,
  },
  {
    ID: 30574063,
    number: 45232502,
    place: Paris,
  },
  ...
];
const arrayItems = [
  {
    "ID": 30574062,
    "anotherNumber": "52,3",
    "color": "red"
  },
  {
    "ID": 30574062,
    "anotherNumber": "13",
    "color": "yellow"
  },
  {
    "ID": 30574063,
    "anotherNumber": "60,6",
    "color": "blue"
  },
  ...
]

//expected result
[
  {
    ID: 30574062,
    number: 28234702,
    place: London,
    data: [
      {
       "anotherNumber": "52,3",
       "color": "red"
      },
      {
       "anotherNumber": "13",
       "color": "yellow"
      }
    ]
  },
  {
    ID: 30574063,
    number: 45232502,
    place: Paris,
    data: [
      {
        "anotherNumber": "60,6",
        "color": "blue"
      },
    ]
  },
  ...
]

CertainPerformance · Answer 1 · 2022-05-30T14:52:51.973

0

There's probably a nicer-looking solution, but perhaps you could stringify the array to be written in chunks, and only generate strings and call appendFile in those chunks, so that at no point there's a single huge string. After creating composed, create a write stream (to allow for many sequential writes without erroring) and keep calling .write on it.

const stream = fs.createWriteStream(filePath, { flags:'a' });
stream.write('[');
const CHUNK_LENGTH = 500; // alter as needed
for (let i = 0; i < composed.length; i += CHUNK_LENGTH) {
  const chunkStr = JSON.stringify(
    composed.slice(i, i + CHUNK_LENGTH)
  );
  stream.write(chunkStr);
}
stream.write(']');

// to wait for all of this to complete, watch for the stream's finish event

edited May 30 '22 at 14:52

answered May 30 '22 at 14:46

CertainPerformance

356,069
52
309
320

Still the same error. Could this code be used with fs.createReadStream on arrayMain and then using filter() on each chunk? – Mogma May 30 '22 at 15:41
Alter the CHUNK_LENGTH as needed - if stringifying 500 is too much, try 200, or 50, or something like that. It should work at some point - experiment with your data to figure out what that is. `createReadStream` isn't right, because all you're doing is *writing* data, not *reading* data. `.filter` could be used, I guess, but iterating with a `for` loop feels more natural. – CertainPerformance May 30 '22 at 15:50
I did change CHUNK_LENGTH but it still produced the same error.Also are u saying that if i were to just run composed without any fs.writeFile() or stream.write() i wouldn't get this error? – Mogma May 30 '22 at 16:34
The only part of the code in your question where you're potentially creating a large string is when you do `JSON.stringify(composed, null, 2)`. If you don't stringify the large array (or parts of it), there won't be any large strings to cause errors. When you're using the code in my answer, is the `const chunkStr = JSON.stringify(` still the one causing the problem? It shouldn't be, because that's not a large string if CHUNK_LENGTH is sufficiently small – CertainPerformance May 30 '22 at 16:39
Hmmm this is the error message: `buffer.js:604 slice: (buf, start, end) => buf.utf8Slice(start, end), Error: Cannot create a string longer than 0x3fffffe7 characters ` – Mogma May 30 '22 at 16:48
There are no other places in your code that have the potential to create a large string. If it's not JSON.stringify, trace the error to see which line throws so that the problem can be debugged. – CertainPerformance May 30 '22 at 16:50
Hmm deleting `stream` and running just `composed`, with both arrays in the same file and nothing else still produces this error. – Mogma May 30 '22 at 17:25
`composed` does not create any large strings - or even any strings at all. Your problem sounds like it's not in the code you've posted. – CertainPerformance May 30 '22 at 17:37

Why do i receive `Error: Cannot create a string longer than 0x3fffffe7 characters`

1 Answers1