3

I am trying to synchronize data between to data stores, the source is mssql and the destination is MongoDB. In this syncing process I am getting a memory heap error. I am not sure why this happens and I am fully aware that the following code may not be the best, but for now I am just trying to understand why the allocation error is coming.

I am compiling my code with babel, in development I am just using babel-node.

try {
  const response = await sqlDataStore.findAll({
    attributes: ['id', 'Name'],
  });
  /* eslint no-restricted-syntax: 0 */
  for (const item of response) {
    /* eslint no-await-in-loop: 0 */
    await this.Model.updateOne({}, item, { upsert: true });
  }
} catch (err) {
  console.log(err);
}

If I understand correctly the heap error is caused by the for loop, so that would mean that every await statement is cached in the memory. I would have expected that every await statement is cleared from the memory because I am not assigning it to any variable.

Updated:

Gladly I found already a solution due to another post: Bulk upsert in MongoDB using mongoose

My Code:

  const response = await sqlDataStore.findAll({
    attributes: ['id', 'Name'],
  });

  const bulkUpdate = response.map(doc => ({
    updateOne: {
      filter: { _id: doc.id },
      update: doc.dataValues,
      upsert: true,
    },
  }));

  this.Model.collection.bulkWrite(bulkUpdate);

If someone is using this solution it should be kept in mind that this also could crash for lots amount of data. The solution provided in the other posts suggests that the data should be processed in buckets of 1000 until every document is updated/inserted.

Just for interest and technical understanding I would appreciate an explanation of what exactly I did wrong in my first code.

Matt
  • 620
  • 4
  • 11
  • Just to be sure, does the code run fine with just the first SQL query? I.e., it's not that it's just too huge an amount of data being brought in? – jameslol May 03 '19 at 05:25
  • yea it is definitely fine for the sql query and the amount of data is also not that big there are just 2401 rows. – Matt May 03 '19 at 05:58
  • check this one https://eslint.org/docs/rules/no-await-in-loop – Atul Agrawal May 03 '19 at 06:23
  • Hi I understand that the operations are delayed but that is not my question, the question is if they are indeed stored in the memory and if that is expected behavior even though I am not storing the response in any variable. – Matt May 03 '19 at 06:27
  • I guess this is because your execution is completely blocked the next execution and will be in memory until your entire execution is complete. Check the link I have shared. – Atul Agrawal May 03 '19 at 06:28
  • Your execution should be parallel instead of synchronous and you should collect the result of every await in some variable which will make your execution more memory efficient as your call stack is not getting free after the execution which causing you out of memory exception.It is not about variable memory, it is actually about function call stack which is causing you this exception. – Atul Agrawal May 03 '19 at 06:30
  • Thank you very much that makes everything clear I try to keep that in mind for future references – Matt May 03 '19 at 06:36
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/192751/discussion-between-atul-agrawal-and-matt). – Atul Agrawal May 03 '19 at 06:38

1 Answers1

2

You are getting this because your function call stack is not getting free since they are waiting for other calls to complete its execution.

Since all the call stacks blocked your stack memory and hence after some executions you are getting out of memory exception.

Check this link: https://eslint.org/docs/rules/no-await-in-loop

As you can see your await call blocked in memory for other await to finish and they are returning you the value in one go which is bad in your code.

It is actually you are making a sync call and each sync call waiting for other sync calls to finish and finally, your sync calls get pile up in stack memory and you are getting the exception .

Atul Agrawal
  • 1,474
  • 5
  • 22
  • 41