Using setImmediate to improve JSON.stringify performance of large amounts of data

Question

I have some code that JSON.stringify's an array of objects like so:

const postsCommentsReadyForDB = postsComments.map(postComments => ({
  id: getPostId(postComments),
  comments: JSON.stringify(getComments(postComments)),
}))

However on very large amounts of data (e.g. an array of 6000+ objects), it can take up to 3+ seconds for this to complete because of all the JSON.stringify processing.

I'm not so worried about the time it takes to do this, but I would like to avoid blocking the event loop or any I/O, so I thought I could perhaps use setImmediate to solve this.

Here is my approach to doing this with setImmediate:

const postsCommentsReadyForDB = []

postsComments.forEach(postComments => {
  setImmediate(_ => {
    postsCommentsReadyForDB.push({
      id: getPostId(postComments),
      comments: JSON.stringify(getComments(postComments)),
    })
  })
})

However when I run this it doesn't seem to work and postsCommentsReadyForDB is empty.

Is it possible to use setImmediate in this scenario?

[Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference](https://stackoverflow.com/questions/23667086/why-is-my-variable-unaltered-after-i-modify-it-inside-of-a-function-asynchron), [How do I return the response from an asynchronous call?](https://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call/14220323#14220323) — Andreas, Apr 15 '21 at 07:52
"*This is a new approach*" - do you still have a problem or is this a solution? If yes, you should [post it as an answer](https://stackoverflow.com/help/self-answer). — Bergi, Apr 16 '21 at 08:56
Btw don't use `then` in an `async` (generator) function. Just write `await setImmediatePromise(); yield {…};`. And the generator function seems unnecessary altogether. You could achieve the same thing (probably even more efficient than repeatedly doing `postComments.shift()`) using `for (const postComment of postComments) { await setImmediatePromise(); postsCommentsForDb.push({…}); }` — Bergi, Apr 16 '21 at 08:57

score 0 · Answer 1 · answered Apr 16 '21 at 09:32

Based on Bergi's suggestion, this approach seems to work:

const util = require('util')
const setImmediatePromise = util.promisify(setImmediate)


async function foo(){

  const postsCommentsReadyForDB = []

  for (const postComments of postsComments) {

    await setImmediatePromise()

    postsCommentsReadyForDB.push({
      id: getPostId(postComments),
      comments: JSON.stringify(getComments(postComments)),
    })
  }

}

foo().catch(err => console.error(err))

Using setImmediate to improve JSON.stringify performance of large amounts of data

1 Answers1