How can I modify the array passed to Promise.all

Question

My node.js class is recursively loading files from the file system asynchronously.

Other methods of my class are not valid until all of the files have finished loading, so I defined an array of promises, and do Promise.all() on the array before executing methods that depend on the loading being complete.

As the code discovers each file or directory, it adds a new promise to the array so that the array eventually contains one promise for each directory scanned and each file loaded. I have verified that this is working as designed.

The problem is that Promise.all() seems to only wait on promises that are already in the array when it is called, and does not wait for promises that are added to the array after Promise.all() is called, so the promise returned by Promise.all() is resolved before all of the files have been loaded.

Yes, I did make sure that new promises are added to the array before the current promise is resolved to avoid that race condition.

Assuming that I can't use Promise.all() in this situation, what is the best alternative?

Related: [How to know when all Promises are Resolved in a dynamic “iterable” parameter?](https://stackoverflow.com/q/37801654/4642212). — Sebastian Simon, Jul 25 '20 at 00:11
Thank you. Yes, this is the same problem I have but I could not find this question by searching. — bikeman868, Jul 25 '20 at 00:19
@bikeman868, You should get all the promises a head of time. — Mu-Majid, Jul 25 '20 at 00:24
You cannot get the list of promises ahead of time because it is recursively (and asynchronously) traversing the directory structure — bikeman868, Jul 25 '20 at 00:40

score 0 · Answer 1 · answered Jul 25 '20 at 00:18

0

I think you should traverse the file system's tree first to get all the files' paths and store them in an array, then Promise.all that array.

Something like this:

(i am using lodash)

async function getPaths(dir) {
  const files = [];
  const promises = [];
  const contents = await fs.readdir(dir);

  for (const content of contents) {
    if(_.endsWith(content, '.js')) { // only get JS files
      files.push(path.resolve(dir, content));
    }
    else {
      promises.push(getPaths(path.resolve(dir, content)));
    }

  }
  let subDirFiles = await Promise.all(promises);
  subDirFiles = _.flatten(subDirFiles);

  return _.union(files, subDirFiles); // contain all files paths from a root `dir`
}

answered Jul 25 '20 at 00:18

Mu-Majid

851
1
9
16

I can't do this because the code recursively (and asynchronously) traverses the directory structure. This loading process is kicked off in the constructor, and the other methods of the class are called whilst the loading is in progress. – bikeman868 Jul 25 '20 at 00:44
May i ask what are you trying to achieve exactly? Why some methods of the class are called whilst the loading is in progress ? – Mu-Majid Jul 25 '20 at 00:49
This class loads some files into memory in its constructor, then provides some methods that return information based on the contents of those files. The methods that return information must wait until all of the files have been loaded. – bikeman868 Jul 25 '20 at 00:55
The class is quite big and we have restrictions on pasting code on public forums. I would have to come up with a contrived example that demonstrates the issue, however my original question explains the situation quite well I think. – bikeman868 Jul 25 '20 at 01:05

score 0 · Accepted Answer · answered Jul 25 '20 at 00:49

Thank you for the link to How to know when all Promises are Resolved in a dynamic "iterable" parameter?. Using info from this thread I was able to come up with a working solution as follows:

  private readonly loadingPromises: Promise<any>[];

  private async finishLoading(): Promise<void> {
    var completedCount = 0;
    while (this.loadingPromises.length != completedCount) {
      var waitCount = this.loadingPromises.length;
      await Promise.all(this.loadingPromises);
      completedCount = waitCount;
    }
    this.loadingPromises.length = 0;
  }

  async somePublicMethod() {
    return this.finishLoading().then(() => {
      //... do stuff here
    });
  }

How can I modify the array passed to Promise.all

2 Answers2