Knex.js: JavaScript heap out of memory

Question

I'm working on a project which uses knex.js for database migration. I tried to query the data and insert it into new table in this way.

    return knex.schema.raw(`select * from transactions`)
    .then(result =>{
        console.log('Migrating transactions')
        return knex('new_transactions').insert(result['rows'])
    })

    .catch((err)=>{
        console.error(err)
    })
}

However, because of the large amount of data in this table, it throws an error like this

<--- Last few GCs --->

[8748:0x102641000]    18957 ms: Mark-sweep 1182.2 (1241.7) -> 1182.2 (1203.2) MB, 1145.3 / 0.0 ms  (average mu = 0.248, current mu = 0.000) last resort GC in old space requested
[8748:0x102641000]    19877 ms: Mark-sweep 1182.2 (1203.2) -> 1182.2 (1203.2) MB, 920.2 / 0.0 ms  (average mu = 0.146, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x3dddbe65be3d]
    1: StubFrame [pc: 0x3dddbe64b476]
Security context: 0x31960631e6e1 <JSObject>
    2: replace [0x3196063105e1](this=0x319706b02201 <Very long string[19409793]>,0x31964e437719 <JSRegExp <String[9]: (\\*)(\?)>>,0x31964e437751 <JSFunction (sfi = 0x3196d83747d1)>)
    3: query [0x3196d83070d9] [/Users/grace/node_modules/knex/lib/runner.js:~128] [pc=0x3dddbe6ea4fc](this=0x3196737add89 <Runner map = 0x31...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x10003b125 node::Abort() [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 2: 0x10003b32f node::OnFatalError(char const*, char const*) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 3: 0x1001a8e85 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 4: 0x1005742a2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 5: 0x10057d7a4 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 6: 0x10054caa6 v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 7: 0x10080b626 v8::internal::Runtime_RegExpExecMultiple(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/grace/.nvm/versions/node/v10.15.0/bin/node]
 8: 0x3dddbe65be3d 
Abort trap: 6

How can I solve this question?

score 1 · Answer 1 · answered Aug 10 '20 at 07:13

You should use INSERT INTO if you would like to insert data from other table, or for loop and process the data with limit and offset, avoid pull the whole table in node js if you are not sure how big will it be.

FYI, to solve js heap out of memory, see this post

Knex.js: JavaScript heap out of memory

1 Answers1