0

I have a directory with many sub-directories, which in turn have multiple sub-directories. Think along the lines of

vvv
  example1
    plugin1
    plugin2
  example2
    plugin1
    plugin2
    plugin3
  etc.

I want to determine how many times a plugin name exists, and add the plugin to an array of objects if it hasn't already been added. For the case where it has in fact been added, each plugin object contains a key "count" with a numeric value representing how many times the plugin folder name has been found:

[
  {name: "plugin1", count: 5},
  {name: "plugin2", count: 2},
  etc.
]

I get that this is likely fairly easy, but I cannot come up with proper, dry code past pushing the first instance of a plugin name to an array. I am reading directories from a file system:

var http = require("http");
var fs = require("fs");

http
  .createServer(function (req, res) {    
    const main = fs.readdirSync("./vvv");
    const newArr = [];

    main.forEach((dir) => {
      const plugins = fs.readdirSync(`./vvv/${dir}`);

      plugins.forEach((plugin, index) => {
        newArr.push({ name: plugin, count: 0 });
      });
    });

    console.log(newArr, "newArr");

    res.end();
  })
  .listen(3000, () => {
    console.log("Server is running at port 3000...");
  });

I think the way would be a reduce function to keep the code as dry as possible, but not sure how to get there.

Codesandbox is giving me readDir trouble, so not sure how to provide an easy working example of the above base-code.

Thank you for helping out.

HJW
  • 1,012
  • 2
  • 13
  • 32

1 Answers1

1

You are adding each entry as unique, not filtering existing ones.

Note: As in initial question, following examples lack error handling. Non async examples (all but the last) would need try {} catch(err) {} blocks. Async examples may use them as well, or Promise.prototype.catch chained after then blocks.

You might use an object instead

    const main = fs.readdirSync("./vvv");
    const newArr = {}; // Object, not array

    main.forEach((dir) => {
      const plugins = fs.readdirSync(`./vvv/${dir}`);

      plugins.forEach((plugin, index) => {
        if(plugin in newArr) {
          newArr[plugin]++;
        }
        else {
          newArr[plugin] = 1;
        }
      });
    });

Or use Array.find

    const main = fs.readdirSync("./vvv");
    const newArr = [];

    main.forEach((dir) => {
      const plugins = fs.readdirSync(`./vvv/${dir}`);

      plugins.forEach((plugin, index) => {
        const found = newArr.find(p => p.name === plugin);
        if(found) {
          found.count++;
        }
        else {
          newArr.push({ name: plugin, count: 1 });
        }
      });
    });

You can use Array.reduce

    const main = fs.readdirSync("./vvv");
    const newArr = main.reduce((acc, dir) => {
      const plugins = fs.readdirSync(`./vvv/${dir}`);

      plugins.forEach((plugin, index) => {
        const found = acc.find(p => p.name === plugin);
        if(found) {
          found.count++;
        }
        else {
          acc.push({ name: plugin, count: 1 });
        }
      });

      return acc;
    }, []);

You can use double Array.reduce and object, immutable

    const main = fs.readdirSync("./vvv");
    const newArr = main.reduce((acc, dir) => (
      fs.readdirSync(`./vvv/${dir}`)
        .reduce((accInner, plugin) => ({
           // Return all already collected plugins
           ...accInner,
           // For the currently handled plugin...
           [plugin]: (plugin in accInner)
             // If already in othe object, return its value + 1
             ? accInner[plugin] + 1
             // If not, create it with a 1 counter
             : 1,
         }), acc) // Initialize accInner to acc
    ), {}); // Object, not array

You can use double Array.reduce and Array.find, immutable

    const main = fs.readdirSync("./vvv");
    const newArr = main.reduce((acc, dir) => (
      fs.readdirSync(`./vvv/${dir}`)
        .reduce((accInner, plugin) => (
           accInner.find(p => p.name === plugin)
             // If a plugin with that name is found in inner accumulator,
             // return a map of that accumulator, modifiying the current
             // plugin (count+1) and passing through the rest
             ? accInner.map(p =>
                 p.name === plugin
                   ? {...p, count: p.count + 1}
                   : p
               )
             // If none is found, return a new array with all the 
             // collected plugins, plus the new one
             : [...accInner, {name: plugin, count: 1}]
         ), acc) // Initialize accInner to acc
    ), []); // Array

You can use double Array.reduce and Array.find, immutable, using async/await and async version on fs

In this example, the workflow should:

  • Launch and wait for resolution of main's readdir: we cannot do anything until having its results
  • Launch all readdir simultaneously/in parallel. All IO operations start their job here.
  • Use reduce technique for resolving promises sequentially. Waits until the first main's array element readdir promise is fulfilled to start parsing all promises results, some of which may need some extra waiting.
  • Use reduce to compute values as in the previous example.
var http = require("http");
var fs = require("fs").promises;

// You can only run async code (using `await` keyword) inside an async function
// You can use use `await` without an extra function declaring your package as a module in `package.json`

const getPluginsUsage = async () => {
  const main = await fs.readdir("./vvv");
  // Wait for all the `then` chain to fulfill
  const newArr = await main
    // Generate an array of promises from the `main` array of dirs
    // All IO operations start here, concurrently/in parallel
    .map( dir => fs.readdir(`./vvv/${dir}`))
    .reduce((prev, promise) => ( 
      // Once initial/previous promise fulfills,
      // add current promise to the `then` chain.
      //  - Get `acc` from previous promise resolution
      //  - Get `plugins` from current promise resolution

      // Note: Async functions return a promise
      //       resolving to its return value
      // This block could also be written as:
      // async (prev, promise) => {
      //   const acc = await prev;
      //   const plugins = await promise;
      //   return plugins.reduce(...);
      // }

      // Or:
      // prev
      //   .then( acc => promise.then( plugins => [acc, plugins] ))
      //   .then( ([acc, plugin]) => plugins.reduce(..) ) 

      // Or:
      // async (prev, promise) => {
      //   const [acc, plugins] = await Promise.all([prev, promise]);
      //   return plugins.reduce(...);
      // }

      prev.then( acc => promise.then( plugins => ( 
        // Parse results as in the previous example
        plugins.reduce( (accInner, plugin) => ( 
          acc.find(p => p.name === plugin)
            ? accInner.map(p =>
                p.name === plugin
                  ? {...p, count: p.count + 1}
                  : p 
              ) 
            : [...accInner, {name: plugin, count: 1}]
        ), acc)
      )))

    // Initial reduce value is a Promise that will resolve
    // to our accumulator base (here, an empty Array)
    ), Promise.resolve([]));

  // Async functions always return a promise
  // This one will resolve to newArr value
  return newArr;
} 

// Using node module package will probably make it easier to handle the http server (this next part is not tested)
getPluginsUsage().then( newArr => {
  http
    .createServer(function (req, res) {    
      console.log({newArr});
      res.end();
    })
    .listen(3000, () => {
      console.log("Server is running at port 3000...");
    });
});

If you feel wild, you can also try Map, WeakMap, Set and WeakSet.

Edit

Further JavaScript readings:

General programming concepts:

emi
  • 2,786
  • 1
  • 16
  • 24
  • Thank you so much for your detailed solutions and explanation. I found the first two methods excellent and so easy that I'm embarrassed that I couldn't get this by myself. The only thing to add is that I believe I'd have to start the count at `newArr[plugin] = 1;` for `You might use an object instead`, and `newArr2.push({ name: plugin, count: 1 });` for the `Array.find` method, rather than 0, otherwise I'm down by one in the total count. I couldn't get any reduce methods to work. Think the wrong folder is being considered, but also `dir` remains undefined even if adjusted for right folders. – HJW Jan 19 '21 at 02:49
  • You are right about `0` and `1`. Just fixed all the answers. – emi Jan 19 '21 at 11:53
  • Tested all of them. There was a problem in the 3rd solution (the first with `reduce`), where the `return acc` was misplaced. Now fixed. – emi Jan 19 '21 at 12:25
  • I tested with Node v12.18.2 (Linux). Which version and operating system are you using? – emi Jan 19 '21 at 12:50
  • I don't know if it was your fix or something on my end but now all of them work like a charm. Thanks a lot for your time and effort. If I wanted to get a better understanding about the concepts you've employed here, what topic should I be looking into? What structural/architectural problem are you addressing here? – HJW Jan 19 '21 at 13:21
  • Edit: added references to global programming concepts and specific for JavaScript. – emi Jan 19 '21 at 14:15
  • I can also add an example taking advantage of `async`/`await` using non-sync `fs` method, allowing to accomplish the task faster when the plugins tree is very large (or just for fun and educational purposes). – emi Jan 19 '21 at 14:18
  • For completeness I'd like to encourage you to add the `async`/`await` example as well. Although I use them all the time I think if others found this post it would add a lot of value. I'll buy you a beer regardless. – HJW Jan 19 '21 at 16:00
  • Just added the `async` response ;) – emi Jan 19 '21 at 17:35