Reading files in multiple directories, matching filenames with their data using Node and Promises

Question

I have an array of directories.

var directories = ['/dir1', '/dir2'];

I want to read all files under these directories, and in the end have an object that matches filenames with their base64/utf8 data. In my case these files are images. The resulting object might look like:

var result = {
 '/dir1': {
  'file1': 'iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAIAAACx0UUtAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWF...',
  'file2': 'iVBORw0KGgoAAAANSUhEUgAAAOEAAADhCAIAAACx0UUtAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWF...'   
 }
}

I easily implemented this with a callback hell, but when I try with Promises, I'm not sure how to pass directory information, and filename to succeeding then() and map() functions.

In the example below I'm using Bluebird library:

const Promise = require('bluebird'),
  fs = Promise.promisifyAll(require('fs'));

var getFiles = function (dir) {
      return fs.readdirAsync(dir);
  };

Promise.map(directories, function(directory) {
    return getFiles(directory)
  }).map(function (files) {
    // which directory's files are these?
    return files;
  })

The next step would be iterating over files and reading their data.

I don't mind if answer is with ES6, Bluebird, or Q.

@DenizOzger are you interested in recursive path resolution and nested directories? — Benjamin Gruenbaum, Apr 29 '15 at 16:50
@BenjaminGruenbaum Yes, as long as I can map filedata to filename and filename to directories, anything works. — Deniz Ozger, Apr 29 '15 at 16:54
@DenizOzger how do you feel about a solution using completely flat flow control with generators? (with `--harmony` although it's completely stable), would that interest you? — Benjamin Gruenbaum, Apr 29 '15 at 16:59
@DenizOzger as you wish, I'm adding an ES7 answer, you can run it with BabelJS (looks like my code :)) — Benjamin Gruenbaum, Apr 29 '15 at 17:27

score 2 · Accepted Answer · edited Apr 30 '15 at 05:30

2

The easiest way with Bluebird would be to use props:

function getFiles(paths) { // an array of file/directory paths
    return Promise.props(paths.reduce(function (obj, path) {
        obj[path.split("/").pop()] = fs.statAsync(path).then(function (stat) {
            if (stat.isDirectory())
                return fs.readdirAsync(path).map(function (p) {
                    return path + "/" + p;
                }).then(getFiles);
            else if (stat.isFile())
                return fs.readFileAsync(path);
        }).error(function (e) {
            console.error("unable to read " + path + ", because: ", e.message);
            // return undefined; implied
        });
        return obj;
    }, {}));
}

edited Apr 30 '15 at 05:30

thefourtheye

233,700
52
457
497

answered Apr 29 '15 at 16:06

Bergi

630,263
148
957
1,375

Got ```TypeError: Cannot read property 'then' of undefined``` on third line. Are our fs definitions the same? ie. ```fs = Promise.promisifyAll(require('fs'))``` – Deniz Ozger Apr 29 '15 at 16:19
Yes, and that should be a `fs.statAsync` of course :-) – Bergi Apr 29 '15 at 16:20
of course :) I passed this array: ```['/dir1', '/dir2']``` to the function. In the first execution of .reduce() callback, obj is ```{}```, path is ```'/dir1'```. In the second execution, obj is ```undefined```, and path is ```'/dir2'```. This ends up in ```Cannot set property '/dir2' of undefined``` Isn't the reduce function this one?: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce – Deniz Ozger Apr 29 '15 at 16:32
yeah, I also forgot to `return obj` from the reducer, so that the object is getting built. – Bergi Apr 29 '15 at 16:33
Something something race condition something something shouldn't matter, something something mention it something. – Benjamin Gruenbaum Apr 29 '15 at 16:48
@BenjaminGruenbaum: What do you mean? That object enumeration order is not guaranteed? – Bergi Apr 29 '15 at 16:52
1

I mean that if a file/directory was added during the enumeration it'll break here (if it's in the stat and doesn't exist - because servers change stuff). – Benjamin Gruenbaum Apr 29 '15 at 16:53
@BenjaminGruenbaum: Oh right, didn't consider that. But it should be only affected by file removals, shouldn't it? What do you suggest, swallowing ENOENT operational errors? – Bergi Apr 29 '15 at 16:56
Yes, swallowing ENOENT is exactly what I'd do – Benjamin Gruenbaum Apr 29 '15 at 16:57
Got ```Error: ENOENT, stat 'image.png'```. I think it's because there is no directory on the path, like ```/dir1/image.png``` when reading, but suspiciously execution never hits inside of ```if (stat.isFile())``` so not sure how I'm ever seeing this error.. I get it after all directories are read if it helps.. – Deniz Ozger Apr 29 '15 at 16:57
1

OK with the latest edit of the code this seems to be resolved – Deniz Ozger Apr 29 '15 at 17:01

Lewis · Answer 2 · 2015-04-29T16:56:28.333

1

I have no idea about Bluebird, but this is the ES6 Promise approach.

var fs = require('fs');
var directories = ['/dir1', '/dir2'];
var result = {};
Promise.all(directories.map(function(dir) {
  result[dir] = {};
  return new Promise(function(resolve, reject) {
    fs.readdir(dir, function(er, files) {
      if (er) {
        return reject(er);
      }
      resolve(files);
    });
  }).then(function(files) {
    return Promise.all(files.map(function(file) {
      return new Promise(function(resolve, reject) {
        fs.readFile(dir + '/' + file, function(er, data) {
          if (er) {
            return reject(er);
          }
          result[dir][file] = data.toString('base64');
          resolve();
        });
      });
    }));
  });
})).then(function() {
  console.log(result);
}).catch(function(er) {
  console.log(er);
});

edited Apr 29 '15 at 16:56

answered Apr 29 '15 at 15:50

Lewis

14,132
12
66
87

1

Don't use the [promise constructor antipattern](http://stackoverflow.com/q/23803743/1048572)! – Bergi Apr 29 '15 at 15:55
Thanks a lot for the answer @Tresdin, I understand how it works and appreciate the ES6 approach, but I'm kinda disappointed as it looks like the callback hell I am trying to avoid. – Deniz Ozger Apr 29 '15 at 16:02
@DenizOzger Yeah, me too. I actually prefer `async` module since it's faster and easier to read. :) – Lewis Apr 29 '15 at 16:21
Wow, I'm so happy I'm using bluebird right now. This is much nicer with bluebird. – Benjamin Gruenbaum Apr 29 '15 at 16:49

Reading files in multiple directories, matching filenames with their data using Node and Promises

2 Answers2