asynchronous counting totals

Question

I have a gulp task that loops through a set of files and validates each one. At the end, I want to report the total number of valid files.

const fs = require('fs')
const path = require('path')
const gulp = require('gulp')

gulp.task('validate', function (callback) {
  function validate(json) { /*...*/ }
  let files = ['file1.json', 'file2.json', 'file3.json']
  let count = 0

  files.forEach(function (file) {
    fs.readFile(path.join(__dirname, file), 'utf8', function (err, data) {
      let is_valid = validate(JSON.parse(data))
      if (!is_valid) {
        console.error(`Invalid! ${file}`)
      } else {
        console.log(`Valid: ${file}`)
        count++
      }
    })
  })

  console.log(`Total valid files: ${count}`)
})

When running this, my console reports:

Total valid files: 0
Valid schema: file1.json
Valid schema: file2.json
Valid schema: file3.json

I suspect the count is reported before the files are read, due to the asynchronous nature of fs.readFile(). When I try logging the total count before .forEach() is called, I get the same result (as expected). How do I log a global variable after an asynchronous function?

Note: I know the gulp convention is to pass a callback argument to the gulp task, but gulp calls this task with a default callback function that can only be passed an Error. Can I somehow create my own custom callback function, to which I could pass count?

Possible duplicate of [How do I return the response from an asynchronous call?](https://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call) — raina77ow, Jan 09 '18 at 21:33
Move the console.log to the line right before or after `count++` — Kevin B, Jan 09 '18 at 21:34
`forEach` is synchronous whereas `fs.readFile` is asynchronous -- not a good mix. Use promises or [async.js](http://caolan.github.io/async/) to know when all operations are done. — Mikey, Jan 09 '18 at 21:43
Edited answer to incorporate **gulp** conventions. How do I use promises and/or custom callback functions with gulp? — chharvey, Jan 09 '18 at 23:47

score 1 · Answer 1 · answered Jan 09 '18 at 23:30

The problem is that 'fs' library is old and uses only callbacks.

The solution I recommend is to promisify the fs.readFile() method and add those promises in array and promise.all them.

Side node: use forof instead of forEach when you use async functions.

const { promisify } = require('util');
const fs = require('fs');
const readFileAsync = promisify(fs.readFile);

const path = require('path');

let files = ['file1.json', 'file2.json', 'file3.json'];
let count = 0
let tasks = [];

function validate(json) { /*...*/ }

for (let file of files) {
    tasks.push(readFileAsync(path.join(__dirname, file), 'utf8')
        .then(data => {
            let is_valid = validate(JSON.parse(data))
            if (!is_valid) {
                console.error(`Invalid! ${file}`)
            } else {
                console.log(`Valid: ${file}`)
                count++
            }
        })
        .catch(err => console.log(err))
    );
}

Promise.all(tasks)
    .then(res => console.log(`Total valid files: ${count}`))
    .catch(err => console.log(err));

well, no, the problem is the api available wasn't being used in a way that accomplishes the goal intended. — Kevin B, Jan 09 '18 at 23:33

asynchronous counting totals

1 Answers1