Array doesnt have expected value - Nodejs

Question

I am running a script which looks into a directory and lists files, then checks for the file type to process, if the extension matches then the file is read and each line of the file (.col which is just a txt file renamed) is inserted into an array.

Now after the file is read and the array is populated I would like to use the array and do some further processing, e.g create a db record. I am missing something really basic here because on each console log I do as below I always get the full items (in my array) of the contents of all files.

So to make it a bit simpler: array is empty. Then file is read and processed and array now has array[0]=line 0 of file array[0]=line 1 of file etc

    const fs = require('fs');
const readline =require('readline');
var files = fs.readdirSync('/home/proj/data');
var path = require('path');
var model=[];
var lineReader=[];

for(var i=0; i<files.length; i++) {

    if(path.extname(files[i]) === ".col") {
       lineReader[i] = readline.createInterface({
            input: require('fs').createReadStream(files[i])
         });  

        lineReader[i].on('line', function (line) {  
          model.push(line); 
        }).on('close', async function() {
          console.log(model); 
      });

}

}

Instead the script is run and array[] holds all lines of all files that match the extension.

Your help is greatly appreciated and anyone is allowed to scorch my JS as I am pretty sure I am missing something basic here.

You will need to use promise or asych/await, here is the link to existing answer which might help you https://stackoverflow.com/questions/54034648/node-js-read-file-using-async-await — WebSwami, Mar 04 '19 at 11:11
What Node.js version do you use? Do you need this preparation to be non-blocking (async) or can it be sync as a first program stage before the server can respond? — vsemozhebuty, Mar 04 '19 at 17:32
I use Node.js V8.10.0, the preparation can be synchronous I dont have an issue there. Thanks for your interest in my issue! — Numant Santz, Mar 04 '19 at 18:30
What are the sizes of your files? Is it feasible to read them one by one as a whole and to just split into the array of lines? — vsemozhebuty, Mar 04 '19 at 21:11
I've added a variant for sync processing of not very big files. Let me know if this is not the appropriate way) — vsemozhebuty, Mar 04 '19 at 21:40

score 2 · Answer 1 · answered Mar 04 '19 at 11:24

So, you want to read the files in parallel (because that's what your program does) and put it in an array of arrays? You can make the reading file mechanism a promise and use it using Promise.all. Here is an example to get you started.

const fs = require('fs');
const readline = require('readline');
var files = fs.readdirSync('./');
var path = require('path');

function readFile(fileName) {
    return new Promise(resolve => {
        const array = [];
        const lineReader = readline.createInterface({
            input: fs.createReadStream(files[i])
        });
        lineReader.on('line', function (line) {
            array.push(line);
        }).on('close', async function () {
            //do some proc
            console.log(array);
            resolve(array);
        });
    });
}
const readFilePromises = [];
for (var i = 0; i < files.length; i++) {
    if (path.extname(files[i]) === ".js") {
        readFilePromises.push(readFile(files[i]));
    }
}

Promise.all(readFilePromises) //or await Promise.all([..])
    .then(data => {
        console.log(data);//will be array of arrays
    })

If you want a single Array you can always flatten the result using data.flat()

Thank you for your input. Actually I do not want to read files in parallel. I would like to prepare my array and then send it to the database in an async function. The catch here is that I was originally using 1 array and resetting it (array=[];) each time the file extension condition is met, so in my mind I would have a clean array each time I read a new .col file. Another issue I did not mention is that this is part of a controller of a web app using sailsjs. — Numant Santz, Mar 04 '19 at 13:15

score 1 · Accepted Answer · answered Mar 04 '19 at 21:37

If your files are not very big and sync methods are OK, you can simplify the code this way:

'use strict';

const fs = require('fs');
const path = require('path');

const model = [];

fs.readdirSync('/home/proj/data')
  .filter(name => path.extname(name) === '.col')
  .forEach((name) => {
    model.push(...fs.readFileSync(name, 'utf8').split('\n'));
  });

console.log(model);

Array doesnt have expected value - Nodejs

2 Answers2