Background
I'm building out a NodeJS application that will capture data from one source database and store these snapshots as JSON in a Postgres DB. For the most part, this is working great, as this removes the overhead of managing the storage tables which can change over time ever so slightly, based upon what SQL is run on the source to capture this data. In essence, take the data and dump in JSONB until needed later on in the process.
Issues
When running some of the larger queries against the source DB (Oracle EBS) , the column length is significant, likewise with the rowcount. ( 150+ columns, 250k+ rows). Although this is pulled in fine to NodeJS and stored as JSON, I hit an issue when trying to parse this for storage in Postgres which is:
RangeError: Invalid string length
at JSON.stringify (<anonymous>)
at prepareObject (/***/node_modules/pg/lib/utils.js:81:15)
at prepareValue (/***/node_modules/pg/lib/utils.js:66:12)
at prepareValueWrapper (/***/node_modules/pg/lib/utils.js:182:12)
at writeValues (/***/node_modules/pg-protocol/dist/serializer.js:66:41)
at Object.bind (/***/node_modules/pg-protocol/dist/serializer.js:97:5)
at Connection.bind (/***/node_modules/pg/lib/connection.js:161:26)
at Query.prepare (/***/node_modules/pg/lib/query.js:204:18)
at Query.submit (/***/node_modules/pg/lib/query.js:155:12)
at Client._pulseQueryQueue (/***/node_modules/pg/lib/client.js:481:45) {stack: 'RangeError:
Invalid string length
at JSON…gration/node_modules/pg/lib/client.js:481:45)', message: 'Invalid string length'}
arg0:RangeError: Invalid string length
at JSON.stringify (<anonymous>)
at prepareObject (/***/node_modules/pg/lib/utils.js:81:15)
at prepareValue (/***/node_modules/pg/lib/utils.js:66:12)
at prepareValueWrapper (/***/node_modules/pg/lib/utils.js:182:12)
at writeValues (/***/node_modules/pg-protocol/dist/serializer.js:66:41)
at Object.bind (/***/node_modules/pg-protocol/dist/serializer.js:97:5)
at Connection.bind (/***/node_modules/pg/lib/connection.js:161:26)
at Query.prepare (/***/node_modules/pg/lib/query.js:204:18)
at Query.submit (/***/node_modules/pg/lib/query.js:155:12)
at Client._pulseQueryQueue (/***/node_modules/pg/lib/client.js:481:45) {stack: 'RangeError:
Invalid string length
at JSON…g...
message:'Invalid string length'
Reading what's on here already I seem to be hitting something like the below whereby I'm reaching the max of what JSON.Stringify() can handle, however, what's called out is pretty old. I've also researched JSONStream, however, my problem is that Postgres still calls Stringify() as a function so anything I do seems to get ignored anyway.
Is there a way I can pass this object directly to Postgres without stringify being called? Or is there a way to chunk this data up and append the column data?
RangeError: Invalid string length --- it should be saying Out Of Memory #14170 JSON.stringify throws RangeError: Invalid string length for huge objects
