How to traverse directory tree in parallel?

Question

First i want to describe the problem: i have a directory tree (depth = 3) which contains several directories and files. Some of those files have .txt extension and some .mp4. I want to copy only .mp4 files in new directory with same hierarchy as it was in source directory (in other words, i don't want to copy all mp4 files in one folder, i want to copy all directories as it is and then copy mp4 files). The question is: how to copy those files not in series, but in parallel? Here is my code:

const fs = require('fs');
const path = require('path');
const { promisify } = require('util');

const sourceDir = process.argv[2];
const stat = promisify(fs.stat);
const copy = promisify(fs.copyFile);
const mkdir = promisify(fs.mkdir);
const readdir = promisify(fs.readdir);
const targetDir = path.join(__dirname, 'only-mp4');

// creating root folder, all files will be copied here
(async () => {
  await mkdir(targetDir);
})();

const copyMediaFiles = async (node) => {
  try {
    const stats = await stat(node);
    if (stats.isDirectory()) {
      let children = await readdir(node);
      // constructing new paths
      children = children.map((child) => path.join(node, child));
      // "copying" file hierarchy (basically just recreating same file hierarchy in target directory)
      children.forEach((child) => {
        const courseDirs = child.split('/').slice(4, 7).join('/');
        mkdir(path.join(targetDir, courseDirs), { recursive: true });
      });
      // running this function for all children recursively in parallel
      const promises = children.map(copyMediaFiles);
      await Promise.all(promises);
    }
    const ext = path.extname(node);
    const filename = path.basename(node);
    // if file extension == mp4 then copy that file in target directory
    if (ext === '.mp4') {
      await copy(
        node,
        path.join(
          targetDir,
          path.dirname(node).split('/').slice(4).join('/'),
          filename
        )
      );
      console.log('File copied: ', filename);
    }
    return;
  } catch (error) {
    console.log(error);
  }
};

copyMediaFiles(sourceDir).then(() => console.log('All mp4 files copied'));

Yes, it's working, but i'm not sure i did it right. Any advise one that? What i did wrong here? And i'm not sure i'm traversing this tree right.

Answer 1

Two issues:

• The first call of copyMediaFiles will happen before the first mkdir promise resolves. This is risky, as you may actually try to access the target directory before its creation completed. If you would just put the call of copyMediaFiles inside the async IIFE, then you don't have that risk:

  (async () => {
      await mkdir(targetDir);
      await copyMediaFiles(sourceDir);
      console.log('All mp4 files copied');
  })();
  

• The second call of mkdir is made without capturing the promise it returns, so a similar risk occurs there as well.

A possible improvement to "crunch" a bit more:

Your aim is to minimise JavaScript idle time (waiting for promises to resolve), and that you can do by maximising the number of promises that are pending.

For that reason it would be better to initiate a call to copyMediaFiles right after the corresonding mkdir promise resolves, and not to initiate the creation of all sibling dirs first before making that call:

const children = await readdir(node);
const promises = children.map(async child => {
    child = path.join(node, child);
    const courseDirs = child.split('/').slice(4, 7).join('/');
    await mkdir(path.join(targetDir, courseDirs), { recursive: true });
    await copyMediaFiles(child);
});
await Promise.all(promises);

With this code you will potentially start a copyMediaFiles call before all sibling directories have been created. If your directories have a high branching factor, then this means you will get a longer list of pending promises, which could be beneficial to the overall performance.

All depends on how well the underlying API manages concurrency.