The sequence of async function inside for loop

Question

Consider the following for-loop

import * as fs from 'fs'

function listAllJs() {
  let files = [ 'abc.js', 'bcd.js', 'e', 'main.js', 'maincopy.ts', 'mainfixedbug.ts', 'package-lock.json', 'package.json' ]
    for (let i = 0; i<files.length; i++) {
      let file = files[i]
      fs.stat(file, (err, stat) => {
        console.log(i, file)
      });
    }
  
}

listAllJs()

The terminal will print out

1 bcd.js
0 abc.js
4 maincopy.ts
2 e
3 main.js
6 package-lock.json
5 mainfixedbug.ts
7 package.json

Or

0 abc.js
3 main.js
2 e
1 bcd.js
4 maincopy.ts
5 mainfixedbug.ts
6 package-lock.json
7 package.json

Or other possible combination like 1 3 2 4 6 7 5

The index is not in ascending order, and the terminal will print out different sequence.
But when i add a console.log(i) before the async function

function listAllJs() {
  let files = [ 'abc.js', 'bcd.js', 'e', 'main.js', 'maincopy.ts', 'mainfixedbug.ts', 'package-lock.json', 'package.json' ]
    for (let i = 0; i<files.length; i++) {
      let file = files[i]
      console.log(i)
      fs.stat(file, (err, stat) => {
        console.log(i, file)
      });
    }
}

The list will always be listed in ascending order.

0
1
2
3
4
5
6
7
0 abc.js
1 bcd.js
2 e
3 main.js
4 maincopy.ts
5 mainfixedbug.ts
6 package-lock.json
7 package.json

I know this kind of question is quite useless but i am learning async function and I really want to know the reason behind it. Any explaination please?

Answer 1

There are no guarantees about how long it will take for fs.stat to run and trigger the callback function.

When your loop only calls it, the various calls come very close together, and the periods of time when fs.stat is running overlap. Sometimes one that starts later will finish sooner.

When you add a console.log statement you make each loop take very slightly longer.

This happens — in your specific test case, on your computer, while your computer is under the load you are tasting it — to make the time it takes to complete a circuit of the loop to be slightly longer than the time it takes fs.stat to get the data.

Since the calls to fs.stat are spaced further apart, they happen to complete in order.

---

You cannot depend upon this behaviour.

---

If you want to have them return in order then:

1. Mark listAllJs with the async keyword.
2. Wrap your call to fs.stat in a function that returns a promise. The promise should resolve to the value you want, and you shouldn't log the value in the callback
3. await the promise and collect its returned value and log that.

Such

function getStat(file) {
    return new Promise( (res, rej) => {
       fs.stat(file, (err, stat) => {
         res(file);
       });
    };
}

function listAllJs() {
  let files = [ 'abc.js', 'bcd.js', 'e', 'main.js', 'maincopy.ts', 'mainfixedbug.ts', 'package-lock.json', 'package.json' ]
    for (let i = 0; i<files.length; i++) {
      let file = files[i];
      const stat = await getStat(file);
      console.log(i, stat)
    }
}

---

Alternatively, to be faster, let the calls to fs.stat run in parallel and store the promises in an array to keep the order. Use Promise.all to read the results when they are all in.

function listAllJs() {
  let files = [ 'abc.js', 'bcd.js', 'e', 'main.js', 'maincopy.ts', 'mainfixedbug.ts', 'package-lock.json', 'package.json' ];
    const promises = [[];
    for (let i = 0; i<files.length; i++) {
      let file = files[i];
      const stat = getStat(file);
      promises.push(stat);
    }
    const stats = await Promise.all(promises);
    stats.forEach( (stat, index) {
        console.log(index, stat)
    });
}

Answer 2

"stat" is running in the background and your program is continued without waiting for it to finish. Therefore multiple instances of it run at the same time, making it unpredictable which one will print the output first. Probably your " console.log(i)" takes long enough to determine which finishes first.

Answer 3

fs/promises and fs.Dirent

Here's an efficient, non-blocking ls program using Node's fast fs.Dirent objects and fs/promises module. This approach allows you to skip wasteful fs.exist or fs.stat calls on every path -

// main.js
import { readdir } from "fs/promises"
import { join } from "path"

async function* ls (path = ".")
{ yield path
  for (const dirent of await readdir(path, { withFileTypes: true }))
    if (dirent.isDirectory())
      yield* ls(join(path, dirent.name))
    else
      yield join(path, dirent.name)
}

async function* empty () {}

async function toArray (iter = empty())
{ let r = []
  for await (const x of iter)
    r.push(x)
  return r
}

toArray(ls(".")).then(console.log, console.error)

Let's get some sample files so we can see ls working -

$ yarn add immutable     # (just some example package)
$ node main.js

[
  '.',
  'main.js',
  'node_modules',
  'node_modules/.yarn-integrity',
  'node_modules/immutable',
  'node_modules/immutable/LICENSE',
  'node_modules/immutable/README.md',
  'node_modules/immutable/contrib',
  'node_modules/immutable/contrib/cursor',
  'node_modules/immutable/contrib/cursor/README.md',
  'node_modules/immutable/contrib/cursor/__tests__',
  'node_modules/immutable/contrib/cursor/__tests__/Cursor.ts.skip',
  'node_modules/immutable/contrib/cursor/index.d.ts',
  'node_modules/immutable/contrib/cursor/index.js',
  'node_modules/immutable/dist',
  'node_modules/immutable/dist/immutable-nonambient.d.ts',
  'node_modules/immutable/dist/immutable.d.ts',
  'node_modules/immutable/dist/immutable.es.js',
  'node_modules/immutable/dist/immutable.js',
  'node_modules/immutable/dist/immutable.js.flow',
  'node_modules/immutable/dist/immutable.min.js',
  'node_modules/immutable/package.json',
  'package.json',
  'yarn.lock'
]

And we want to filter just .js files -

import { extname } from "path"

async function* filter(iter = empty(), test = x => x)
{ for await (const x of iter)
    if (Boolean(test(x)))
      yield x
}

const lsJs = (path = ".") =>
  filter                          // <- filtered stream
    ( ls(path)                    // <- input stream
    , p => extname(p) === ".js"   // <- filter predicate
    )

toArray(lsJs(".")).then(console.log, console.error)
// => ...

[
  'main.js',
  'node_modules/immutable/contrib/cursor/index.js',
  'node_modules/immutable/dist/immutable.es.js',
  'node_modules/immutable/dist/immutable.js',
  'node_modules/immutable/dist/immutable.min.js'
]

A more generic lsExt allows us to filter by any extension. We are not limited to only .js -

const lsExt = (path = ".", ext) =>
  ext
    ? filter(ls(path), p => extname(p) === ext)
    : ls(path)

toArray(lsExt(".", ".json")).then(console.log, console.error)
// => ...

[
  'node_modules/immutable/package.json',
  'package.json'
]

It's hard to debug big functions that take on too many responsibilities. Breaking the problem down made our program easier to write and our functions are highly reusable too. The next step would be to define our set of features in a module. For added explanation and other ways to leverage async generators, see this Q&A.