Reading all files in a directory, store them in objects, and send the object

Question

I do not know if this is possible, but here goes. And working with callbacks makes it even more difficult.

I have a directory with html files that I want to send back to the client in Object chunks with node.js and socket.io.

All my files are in /tmpl

So socket needs to read all the files in /tmpl.

for each file it has to store the data in an object with the filename as the key, and the content as the value.

  var data;
  // this is wrong because it has to loop trough all files.
  fs.readFile(__dirname + '/tmpl/filename.html', 'utf8', function(err, html){
      if(err) throw err;
      //filename must be without .html at the end
      data['filename'] = html;
  });
  socket.emit('init', {data: data});

The final callback is also wrong. It has to be called when all the files in the directory are done.

But I do not know how to create the code, anyone know if this is possibel?

Answer 1

So, there are three parts. Reading, storing and sending.

Here's the reading part:

var fs = require('fs');

function readFiles(dirname, onFileContent, onError) {
  fs.readdir(dirname, function(err, filenames) {
    if (err) {
      onError(err);
      return;
    }
    filenames.forEach(function(filename) {
      fs.readFile(dirname + filename, 'utf-8', function(err, content) {
        if (err) {
          onError(err);
          return;
        }
        onFileContent(filename, content);
      });
    });
  });
}

Here's the storing part:

var data = {};
readFiles('dirname/', function(filename, content) {
  data[filename] = content;
}, function(err) {
  throw err;
});

The sending part is up to you. You may want to send them one by one or after reading completion.

If you want to send files after reading completion you should either use sync versions of fs functions or use promises. Async callbacks is not a good style.

Additionally you asked about stripping an extension. You should proceed with questions one by one. Nobody will write a complete solution just for you.

Answer 2

For all example below you need to import fs and path modules:

const fs = require('fs');
const path = require('path');

Read files asynchronously

function readFiles(dir, processFile) {
  // read directory
  fs.readdir(dir, (error, fileNames) => {
    if (error) throw error;

    fileNames.forEach(filename => {
      // get current file name
      const name = path.parse(filename).name;
      // get current file extension
      const ext = path.parse(filename).ext;
      // get current file path
      const filepath = path.resolve(dir, filename);

      // get information about the file
      fs.stat(filepath, function(error, stat) {
        if (error) throw error;

        // check if the current path is a file or a folder
        const isFile = stat.isFile();

        // exclude folders
        if (isFile) {
          // callback, do something with the file
          processFile(filepath, name, ext, stat);
        }
      });
    });
  });
}

Usage:

// use an absolute path to the folder where files are located
readFiles('absolute/path/to/directory/', (filepath, name, ext, stat) => {
  console.log('file path:', filepath);
  console.log('file name:', name);
  console.log('file extension:', ext);
  console.log('file information:', stat);
});

Read files synchronously, store in array, natural sorting

/**
 * @description Read files synchronously from a folder, with natural sorting
 * @param {String} dir Absolute path to directory
 * @returns {Object[]} List of object, each object represent a file
 * structured like so: `{ filepath, name, ext, stat }`
 */
function readFilesSync(dir) {
  const files = [];

  fs.readdirSync(dir).forEach(filename => {
    const name = path.parse(filename).name;
    const ext = path.parse(filename).ext;
    const filepath = path.resolve(dir, filename);
    const stat = fs.statSync(filepath);
    const isFile = stat.isFile();

    if (isFile) files.push({ filepath, name, ext, stat });
  });

  files.sort((a, b) => {
    // natural sort alphanumeric strings
    // https://stackoverflow.com/a/38641281
    return a.name.localeCompare(b.name, undefined, { numeric: true, sensitivity: 'base' });
  });

  return files;
}

Usage:

// return an array list of objects
// each object represent a file
const files = readFilesSync('absolute/path/to/directory/');

Read files async using promise

More info on promisify in this article.

const { promisify } = require('util');

const readdir_promise = promisify(fs.readdir);
const stat_promise = promisify(fs.stat);

function readFilesAsync(dir) {
  return readdir_promise(dir, { encoding: 'utf8' })
    .then(filenames => {
      const files = getFiles(dir, filenames);

      return Promise.all(files);
    })
    .catch(err => console.error(err));
}

function getFiles(dir, filenames) {
  return filenames.map(filename => {
    const name = path.parse(filename).name;
    const ext = path.parse(filename).ext;
    const filepath = path.resolve(dir, filename);

    return stat({ name, ext, filepath });
  });
}

function stat({ name, ext, filepath }) {
  return stat_promise(filepath)
    .then(stat => {
      const isFile = stat.isFile();

      if (isFile) return { name, ext, filepath, stat };
    })
    .catch(err => console.error(err));
}

Usage:

readFiles('absolute/path/to/directory/')
  // return an array list of objects
  // each object is a file
  // with those properties: { name, ext, filepath, stat }
  .then(files => console.log(files))
  .catch(err => console.log(err));

Note: return undefined for folders, if you want you can filter them out:

readFiles('absolute/path/to/directory/')
  .then(files => files.filter(file => file !== undefined))
  .catch(err => console.log(err));

Answer 3

This is a modern Promise version of the previous one, using a Promise.all approach to resolve all promises when all files have been read:

/**
 * Promise all
 * @author Loreto Parisi (loretoparisi at gmail dot com)
 */
function promiseAllP(items, block) {
    var promises = [];
    items.forEach(function(item,index) {
        promises.push( function(item,i) {
            return new Promise(function(resolve, reject) {
                return block.apply(this,[item,index,resolve,reject]);
            });
        }(item,index))
    });
    return Promise.all(promises);
} //promiseAll

/**
 * read files
 * @param dirname string
 * @return Promise
 * @author Loreto Parisi (loretoparisi at gmail dot com)
 * @see http://stackoverflow.com/questions/10049557/reading-all-files-in-a-directory-store-them-in-objects-and-send-the-object
 */
function readFiles(dirname) {
    return new Promise((resolve, reject) => {
        fs.readdir(dirname, function(err, filenames) {
            if (err) return reject(err);
            promiseAllP(filenames,
            (filename,index,resolve,reject) =>  {
                fs.readFile(path.resolve(dirname, filename), 'utf-8', function(err, content) {
                    if (err) return reject(err);
                    return resolve({filename: filename, contents: content});
                });
            })
            .then(results => {
                return resolve(results);
            })
            .catch(error => {
                return reject(error);
            });
        });
  });
}

How to Use It:

Just as simple as doing:

readFiles( EMAIL_ROOT + '/' + folder)
.then(files => {
    console.log( "loaded ", files.length );
    files.forEach( (item, index) => {
        console.log( "item",index, "size ", item.contents.length);
    });
})
.catch( error => {
    console.log( error );
});

Supposed that you have another list of folders you can as well iterate over this list, since the internal promise.all will resolve each of then asynchronously:

var folders=['spam','ham'];
folders.forEach( folder => {
    readFiles( EMAIL_ROOT + '/' + folder)
    .then(files => {
        console.log( "loaded ", files.length );
        files.forEach( (item, index) => {
            console.log( "item",index, "size ", item.contents.length);
        });
    })
    .catch( error => {
        console.log( error );
    });
});

How it Works

The promiseAll does the magic. It takes a function block of signature function(item,index,resolve,reject), where item is the current item in the array, index its position in the array, and resolve and reject the Promise callback functions. Each promise will be pushed in a array at the current index and with the current item as arguments through a anonymous function call:

promises.push( function(item,i) {
        return new Promise(function(resolve, reject) {
            return block.apply(this,[item,index,resolve,reject]);
        });
    }(item,index))

Then all promises will be resolved:

return Promise.all(promises);

Answer 4

Are you a lazy person like me and love npm module :D then check this out.

npm install node-dir

example for reading files:

var dir = require('node-dir');

dir.readFiles(__dirname,
    function(err, content, next) {
        if (err) throw err;
        console.log('content:', content);  // get content of files
        next();
    },
    function(err, files){
        if (err) throw err;
        console.log('finished reading files:', files); // get filepath 
   });    

Answer 5

I just wrote this and it looks more clean to me:

const fs = require('fs');
const util = require('util');

const readdir = util.promisify(fs.readdir);
const readFile = util.promisify(fs.readFile);

const readFiles = async dirname => {
    try {
        const filenames = await readdir(dirname);
        console.log({ filenames });
        const files_promise = filenames.map(filename => {
            return readFile(dirname + filename, 'utf-8');
        });
        const response = await Promise.all(files_promise);
        //console.log({ response })
        //return response
        return filenames.reduce((accumlater, filename, currentIndex) => {
            const content = response[currentIndex];
            accumlater[filename] = {
                content,
            };
            return accumlater;
        }, {});
    } catch (error) {
        console.error(error);
    }
};

const main = async () => {

    const response = await readFiles(
        './folder-name',
    );
    console.log({ response });
};

You can modify the response format according to your need. The response format from this code will look like:

{
   "filename-01":{
      "content":"This is the sample content of the file"
   },
   "filename-02":{
      "content":"This is the sample content of the file"
   }
}

Answer 6

Another version with Promise's modern method. It's shorter that the others responses based on Promise :

const readFiles = (dirname) => {

  const readDirPr = new Promise( (resolve, reject) => {
    fs.readdir(dirname, 
      (err, filenames) => (err) ? reject(err) : resolve(filenames))
  });

  return readDirPr.then( filenames => Promise.all(filenames.map((filename) => {
      return new Promise ( (resolve, reject) => {
        fs.readFile(dirname + filename, 'utf-8',
          (err, content) => (err) ? reject(err) : resolve(content));
      })
    })).catch( error => Promise.reject(error)))
};

readFiles(sourceFolder)
  .then( allContents => {

    // handle success treatment

  }, error => console.log(error));

Answer 7

If you have Node.js 8 or later, you can use the new util.promisify. (I'm marking as optional the parts of the code that have to do with reformatting as an object, which the original post requested.)

  const fs = require('fs');
  const { promisify } = require('util');

  let files; // optional
  promisify(fs.readdir)(directory).then((filenames) => {
    files = filenames; // optional
    return Promise.all(filenames.map((filename) => {
      return promisify(fs.readFile)(directory + filename, {encoding: 'utf8'});
    }));
  }).then((strArr) => {
    // optional:
    const data = {};
    strArr.forEach((str, i) => {
      data[files[i]] = str;
    });
    // send data here
  }).catch((err) => {
    console.log(err);
  });

Answer 8

For the code to work smoothy in different enviroments, path.resolve can be used in places where path is manipulated. Here is code which works better.

Reading part:

var fs = require('fs');

function readFiles(dirname, onFileContent, onError) {
  fs.readdir(dirname, function(err, filenames) {
    if (err) {
      onError(err);
      return;
    }
    filenames.forEach(function(filename) {
      fs.readFile(path.resolve(dirname, filename), 'utf-8', function(err, content) {
        if (err) {
          onError(err);
          return;
        }
        onFileContent(filename, content);
      });
    });
  });
}

Storing part:

var data = {};
readFiles(path.resolve(__dirname, 'dirname/'), function(filename, content) {
  data[filename] = content;
}, function(error) {
  throw err;
});

Answer 9

async/await

const { promisify } = require("util")
const directory = path.join(__dirname, "/tmpl")
const pathnames = promisify(fs.readdir)(directory)

try {
  async function emitData(directory) {
    let filenames = await pathnames
    var ob = {}
    const data = filenames.map(async function(filename, i) {
      if (filename.includes(".")) {
        var storedFile = promisify(fs.readFile)(directory + `\\${filename}`, {
          encoding: "utf8",
        })
        ob[filename.replace(".js", "")] = await storedFile
        socket.emit("init", { data: ob })
      }
      return ob
    })
  }

  emitData(directory)
} catch (err) {
  console.log(err)
}

Who wants to try with generators?

Answer 10

So, If someone is looking for an beginner friendly approach, here is mine.

Before the actual solution, we have to understand asynchronous functions and promises. An asynchronous function run outside the scope of single thread, and can run parallel to the main thread. It basically means, if a function is asynchronous, JavaScript instead of waiting for the function to complete, will go the next line. The function will execute parallely.

fs.readFile() is asynchronous, thus it executes the next line, and run parallely,

Now, let's understand promises. Promise is basically an object which returns either success or failiure of an asynchorous function. For Example:

//Defining Promise
const promise = new Promise((resolve,reject)=>{
    //This is an asynchronous function, which takes 2 seconds to execute
    setTimeout(()=>{
        if(1!=0){
            //If there is an error reject the promise
            reject(new Error("This is an error messahe"))
        } 
        else{
            //if there are no errors we resolve the promise
            resolve({'userId':'id'})
        }
    },2000)
})

Now, coming the the original problem

const fs = require("fs");

// Read the File Names.
function readFileNames() {
  // Defining a new promise
  return new Promise((resolve, reject) => {
    try {
      //read the directory
      fs.readdir("./public/", (err, files) => {
        // If read completes, resolve the promise.
        resolve(files);
      });
    } catch (err) {
      // If there is an error, reject the promise. 
      reject(err);
    }
  });
}

// Read content of a given file
function readFileContent(file) {
  return new Promise((resolve, reject) => {
    try {
      fs.readFile("./public/" + file, "utf8", (err, content) => {
        resolve(content);
      });
    } catch (err) {
      reject(err);
    }
  });
}

//sending the data
module.exports = {
// If we want to wait for a function to wait for a promise to be 
// resolved we define it as 'async'
  async get(req, res) {
    let data = {};
    //Here we wait for the promise to resolve, thus await is used
    const fileNames = await readFileNames();
    let count = 0;
    // Note we again use async since we want to wait for promise
    fileNames.forEach(async (name) => {
      // We wait for the content of file.
      const content = await readFileContent(name);
      data[name] = content;
      // Since we want to send data after the loop is completed.
      if (count === fileNames.length - 1) {
        res.send(data);
      }
      count++;
    });
  }

Answer 11

In this example I create a array. But you can create a object if you want

const fs = require('fs-extra')

const dirname = `${process.cwd()}/test`
const fileNames = await fs.readdir(dirname)

const files = []
for(const fileName of fileNames){
    const file = await fs.readFile(`${dirname}/${fileName}`, 'utf8')
    files.push(file)
}          

Answer 12

As it's 2022 there are now cleaner native functions for this.

import fs from "fs/promises";

const readDirectory = async (dir) => {
  const files = await fs.readdir(dir);

  const values = files.flatMap(async (file) => {
    const filePath = path.join(dir, file);
    const stat = await fs.stat(filePath);
    if (stat.isDirectory()) {
      return;
    }
    return fs.readFile(filePath);
  });

  const buffers = await Promise.all(values);
  // Remove this line to keep the raw buffers
  const contents = buffers.filter(Boolean).map((l) => l.toString());
  return contents;
};

And a typescript version

import fs from "fs/promises";

const readDirectory = async (dir: string) => {
  const files = await fs.readdir(dir);

  const values = files.flatMap(async (file) => {
    const filePath = path.join(dir, file);
    const stat = await fs.stat(filePath);
    if (stat.isDirectory()) {
      return;
    }
    return fs.readFile(filePath);
  }) as Array<Promise<Buffer>>;

  const buffers = await Promise.all(values);
  // Remove this line to keep the raw buffers
  const contents = buffers.filter(Boolean).map((l) => l.toString());
  return contents;
};

and a recursive version

import fs from "fs/promises";

const getRecursiveFileReads = async (dir: string): Promise<Array<Promise<Buffer>>> => {
  const files = await fs.readdir(dir);
  return files.flatMap(async (file) => {
    const filePath = path.join(dir, file);
    const stat = await fs.stat(filePath);
    if (stat.isDirectory()) {
      return getRecursiveFileReads(filePath);
    }
    return fs.readFile(filePath);
  }) as Array<Promise<Buffer>>;
};
const readDirectory = async (dir: string) => {
  const promises = await getRecursiveFileReads(dir);
  const buffers = await Promise.all(promises);
  return buffers;
};