Node.js wait for all files read in the loop

Question

I am new to the javascript/node.js event driven paradigm.

I need to stop the for after forEach to make sure all files have been read and then I continue. How should I implement wait_for_all_files_read() in this context?

my_list.forEach(function(element) {
  fs.readFile(element.file_path, function(err, data)  {
    if(err) throw err;
    element.obj=JSON.parse(data);
  });
});
wait_for_all_files_read(); <-----------
analyze(my_list)

Neither solution [1] or [2] work for me.

RidgeA · Answer 1 · 2018-02-22T10:02:53.383

I can suggest you to rewrite the code to promise - it will make much easy to deal with it

const {promisisfy} = require('util')
const fs = require('fs')
const readFile = promisify(fs.readFile)

const fileNames = getFilenamesSomehow() // <-- returns array with path, e.g. ["./package.json", "/etc/hosts", "/etc/passwd"]

Promise.all(fileNames.map(name => readFile(name)))
.then(arrayWithFilesContent => analyze(arrayWithFilesContent))
.catch(err => handleError(err))

Next step what you may do - move the code to async/await functions

UPD

Assume you need read only one file, then parse it data to json and analyze the result somehow.

It this case you can do next:

readFile(singleFileName)
.then(function (singleFileContent) {
   return JSON.parse(singleFileContent)
})
.then(function (singleFileContentInJson) {
   return analyze(singleFileContentInJson)
})
.catch(funciton (error) {
  //here you can process all errors from functions above: reading file error, JSON.parse error, analyze error...
  console.log(error)
})

Then assume you need to analyze bunch of files

const fileNames = [...] // array with file names
// this create an array of promises, each of them read one file and returns the file content in JSON
const promises = fileNames.map(function (singleFileName) {
    return readFile(singleFileName)
    .then(function (singleFileContent) {
       return JSON.parse(singleFileContent)
    })
})

// promise all resolves (calls callback in 'then') all of promises in array are resolved and pass to then callback array with result of each promise
Promise.all(promises)
  .then(function (arrayWithResults) {
    return analyze(arrayWithResults)
  })
  // catch callback calls if one of promises in array rejects with error from the promise - so you can handle e.g. read file error or json parsing error here
  .catch(function (error) {
    //here you can handle any error
    console.log(error)
  })

Try to google some article to read how does promises work. E. g. you can start form mdn article

can I use `function(name) {...}` instead of `name => readFile(name)`? — ar2015, Feb 22 '18 at 09:12
inside `readFile` I have another callback, does it wait for all the inner callbacks as well? — ar2015, Feb 22 '18 at 09:13
nope. `Promisify` function just wrap function that has been passed as parameter with promise. — RidgeA, Feb 22 '18 at 09:22
but `readFIle` has a callback inside itself. How to consider that? — ar2015, Feb 22 '18 at 09:25
Also, where should I place `if(err) throw err;` and `element.obj=JSON.parse(data);`? — ar2015, Feb 22 '18 at 09:31

Alex · Accepted Answer · 2018-02-22T09:29:59.247

1

How I would do that:

Promisify fs.readFile (by using, for example Bluebird)
Mark the function as async
Make a list of callbacks (my_list.map instead of forEach)
"await Promise.all(myListOfCallbacks)"
Next line after await will be executed after all the operations been finished

Something like that:

const {promisisfy} = require('util')
const fs = require('fs')
const readFile = promisify(fs.readFile)

const fileNames = getFilenamesArray();

async function executeMe() {
  try {
    const arrayWithFilesContent = await Promise.all(
      fileNames.map(name => readFile(name))
    );
    return analyze(arrayWithFilesContent);
  }
  catch (err) {
    handleError(err)
  }
}

executeMe();

edited Feb 22 '18 at 09:29

answered Feb 22 '18 at 08:56

Alex

4,621
1
20
30

but `readFIle` has a callback inside itself. How to consider that? – ar2015 Feb 22 '18 at 09:26
when it's promisified it returns promise. We can use promise .then (or await) instead of providing a callback as an argument. – Alex Feb 22 '18 at 09:28
So, where should I place `if(err) throw err;` and `element.obj=JSON.parse(data);`? – ar2015 Feb 22 '18 at 09:29
you can remove try/catch block, than the error will be thrown by the Promise.all. regarding the elements - you can do it in the loop AFTER the Promise.all – Alex Feb 22 '18 at 09:33
I became very confused. – ar2015 Feb 22 '18 at 09:34
What is not clear for you? I can explain if you want. – Alex Feb 22 '18 at 09:35
in your code, `readFile(name)` has no callback. How does it use the result of reading the file? – ar2015 Feb 22 '18 at 09:36
arrayWithFilesContent will have an array of all the results. You'll get the results in quasi-synchronous way. – Alex Feb 22 '18 at 09:37
const content = await readFile(fileName); // promisified, not the original fs. readFile! – Alex Feb 22 '18 at 09:42
The output, `arrayWithFilesContent`, is `[ – ar2015 Feb 22 '18 at 09:51
You need to specify the encoding. you can try to change the call from readFile(name) to readFile(name, 'utf8') for example (if you have unicode UTF-8 file, of course). – Alex Feb 22 '18 at 09:56
Then, I need to know which output belong to which input so I have to use an iteration and use the index. wouldn't be easier using `var contents = fs.readFileSync('DATA', 'utf8');` – ar2015 Feb 22 '18 at 10:01
It's possible to do it in sync way as well, and if you have something like command line utility I would recommend to do that, but for typical request/response is better to avoid blocking API. – Alex Feb 22 '18 at 10:05

Node.js wait for all files read in the loop

2 Answers2