Javascript background loop

Question

Say we have a loop.js file:

longLoop().then(res => console.log('loop result processing started'))
console.log('read file started')
require('fs').readFile(__filename, () => console.log('file processing started'))
setTimeout(() => console.log('timer fires'), 500)

async function longLoop () {
  console.log('loop started')
  let res = 0
  for (let i = 0; i < 1e7; i++) {
    res += Math.sin(i) // arbitrary computation heavy operation
    if (i % 1e5 === 0) await null /* solution: await new Promise(resolve => setImmediate(resolve)) */
  }
  console.log('loop finished')
  return res
}

Which if ran (node loop.js) outputs:

loop started
read file started
loop finished
loop result processing started
timer fires
file processing started

How can this code be rewritten to read and process file while the loop is running in the background?

My solution

What I came up with is this:

longLoop().then(res => console.log('loop result processing started'))
console.log('read file started')
require('fs').readFile(__filename, () => console.log('file processing started'))
setTimeout(() => console.log('timer fires'), 500)

async function longLoop () {
  let res = 0
  let from = 0
  let step = 1e5
  let numIterations = 1e7
  function doIterations() {
    //console.log(from)
    return new Promise(resolve => {
      setImmediate(() => { // or setTimeout
        for (let i = from; (i < from + step) && (i < numIterations); i++) {
          res += Math.sin(i)
        }
        resolve()
      })
    })
  }
  console.log('loop started')
  while (from < numIterations) {
    await doIterations()
    from += step
  }
  console.log('loop finished')
  return res
}

Which indeed logs:

loop started
read file started
file processing started
timer fires
loop finished
loop result processing started

Is there a simpler, more concise way to do that? What are the drawbacks of my solution?

Is the target platform single-core or multi-core? It matters because, multi core platforms can leverage multi-process solutions that don't hold up the event loop. For single core environment, while your solution seems good enough, could use constructs like `process.nextTick` - all inescapably holding up the event loop (this applies to the above code too). — , Aug 06 '17 at 08:24
If it is single core then your solution is the only way, but I don't think it is working as you expect it to work since it would be single threaded and would process only 1 task at a time — marvel308, Aug 06 '17 at 08:28
@ManasJayanth, @marvel308 Yes the platform is single-core (the most general case), and I'm perfectly aware of `node.js`'s single threaded nature. Can you elaborate more on `process.nextTick`? i don't see how it can be of any help since it adds functions to the head of the event queue (holding up other tasks as you noted). — grabantot, Aug 06 '17 at 08:39
@grabantot setImmediate is holding up events in the cycle of events. Not saying process.nextTick is any better - but it is as valid as setImmediate. Just the slot of CPU time order will differ - giving us a chance to schedule differently :) — , Aug 06 '17 at 08:46
If you really want the `for` loop logic to run in parallel with other processing, then put it in a worker process (which can be node.js or any other tech) and then let the two processes communicate. This will allow the OS to truly time slice them (whether single CPU or multi CPU). — jfriend00, Aug 06 '17 at 09:09
Why are some people here claiming that the number of cores is in any way important? You don't seem to understand how multithreading works. Effectively the only difference between single and multi core systems is the performance. — Tesseract, Aug 06 '17 at 09:36
@jfriend00 Such usage of `async/await` is totally valid. Thanks for mentioning workers, but that's another topic. I was asking this question to get better understanding of event loop and ways to affect it. — grabantot, Aug 06 '17 at 11:06
Reading your above comment, i think [an earlier question of mine](https://stackoverflow.com/q/39184297/4543207) might give you some further info regarding the event loop and the microtasks. — Redu, Aug 06 '17 at 16:00

score 3 · Accepted Answer · answered Aug 06 '17 at 09:17

The reason why the first version of your code blocks further processing is that await gets an immediately resolving promise (the value null gets wrapped in a promise, as if you did await Promise.resolve(null)). That means the code after await will resume during the current "task": it merely pushes a microtask in the task queue, that will get consumed within the same task. All other asynchronous stuff you have pending is waiting in the task queue, not the microtask queue.

This is the case for setTimeout, and also for readFile. Their callbacks are pending in the task queue, and as a consequence will not get priority over the mircrotasks generated by the awaits.

So you need a way to make the await put something in the task queue instead of the microtask queue. This you can do by providing a promise to it that will not immediately resolve, but only resolves after the current task.

You could introduce that delay with .... setTimeout:

const slowResolve = val => new Promise(resolve => setTimeout(resolve.bind(null, val), 0));

You would call that function with the await. Here is a snippet that uses an image load instead of a file load, but the principle is the same:

const slowResolve = val => new Promise(resolve => setTimeout(resolve.bind(null, val), 0));

longLoop().then(res => 
    console.log('loop result processing started'))

console.log('read file started')

fs.onload = () => 
    console.log('file processing started');
fs.src = "https://images.pexels.com/photos/34950/pexels-photo.jpg?h=350&auto=compress&cs=tinysrgb";

setTimeout(() => console.log('timer fires'), 500)

async function longLoop () {
  console.log('loop started')
  let res = 0
  for (let i = 0; i < 1e7; i++) {
    res += Math.sin(i) // arbitrary computation heavy operation
    if (i % 1e5 === 0) await slowResolve(i);
  }
  console.log('loop finished')
  return res
}

<img id="fs" src="">

Thanks. In my first example I have changed `await null` to `await new Promise(resolve => setImmediate(resolve))`. I knew I was close. — grabantot, Aug 06 '17 at 10:54
@grabantot - In my opinion, any code that relies on the difference between something being a microtask queue vs a regular task queue in order to work properly is fragile (when someone else works on it) and is not very explicit as to its intention. IMO, if you want something to happen after something else, then schedule it that way with an appropriate delay rather than relying on minute differences between types of tasks and then your code will be obvious even to people that don't understand the subtle differences in scheduling between a micro task and a regular task. — jfriend00, Aug 06 '17 at 15:31

Javascript background loop

My solution

1 Answers1