1

So I have a bunch of csv files which have more data then I require, I want to filter the data out by only having keeping the rows with dates after year 2015. The Problem is that it works for a single file but when I enter multiple files it writes the same data in all the streams. So can someone help me out...

Here is my code:

const fastcsv = require('fast-csv');

const csv = require('csv-parser');
const fs = require('fs');

const dateList = new Date('2015-01-01')


let files = fs.readdirSync("dir");

for(let file of files){
    var list = []
    
    
    console.log('<inputPath>'+file);
    fs.createReadStream('<inputPath>'+file)
        .pipe(csv())
        .on('data', (row) => {

            //filtering data here
            var dateRow = new Date(row.Date);
            if(dateRow >= dateList)  list.push(row);

        })
        .on('end', () => {

            //my writestream, I dont know if I should make some function or do it here itself

            const ws = fs.createWriteStream('<outputPath>' + file)
                .then(()=>{
                    console.log(`CSV file successfully processed : ${file}`);
                    fastcsv
                        .write(list, { headers: true })
                        .pipe(ws);
                })

    });
}

I am aware that I should use some setTimeout or some callback function but I don't know where exactly

Bhagyesh Ganatra
  • 33
  • 1
  • 4
  • Update: I had to change the coding language itself to do this task, as my final output was a bunch of CSVs itself, I used Python and it worked out pretty well. But I would appreciate if the concept had been cleared :) – Bhagyesh Ganatra Mar 30 '21 at 17:33

0 Answers0