Cannot access global array and write it to csv

Question

I'm new to javascript and especially node and would appreciate some help.

I have the following code which looks up delivery status for various packages. Everything works fine except the final step, which is writing the results to a CSV file using the csv-writer package from npm.

I process each line of the source CSV and populate an array with the results. The array is declared at the top of the file as a const (final_result) outside of any function. In my understanding, this means it can be accessed by all the functions in the code. Is this right?

The problem I'm having is that when I get to the final step (writing the results to a new CSV file), the final_result length is 0 and contains no data !

When I console-log in the first and second functions, there is data showing.I can also see the array being populated as each row is processed

What am I missing or is there a better way (best practice? ) to achieve this. Thanks.

const createCsvWriter = require('csv-writer').createObjectCsvWriter;
const yt = require('yodel-tracking');
const csv = require('csv-parser');
const fs = require('fs');

const final_result = []; // This is showing as empty at the end ?!

const csvWriter = createCsvWriter({
  path: 'out.csv',
  header: [
    { id: 'Location', title: 'Location' },
    { id: 'Ref', title: 'Ref' },
    { id: 'Scan', title: 'Scan' },
    { id: 'Status', title: 'Status' }
  ]
});

//Functions in order of how they are called

//1 of 3
function run() {
  fs.createReadStream('yodeldata.csv')
    .on('error', () => console.log('error'))
    .pipe(csv())
    .on('data', row => {
      output(row); //calling function 2 of 3
    });
}

//2 of 3
function output(row) {
  yt(row['REF'], 'GB', function(err, result) {
    let data = result.parcel_tracking.parcel_status[0];
    if (!err) {
      let obj = {
        Ref: `${row['REF']}`,
        Location: `${data.location}`,
        Scan: `${data.scan_date} ${data.scan_time}`,
        Status: `${data.status_description}`
      };
      final_result.push(obj); //Pushing each result to the array here
      console.log(final_result.length); // I can see the length increasing sequentially here
    }
  });
}

//3 of 3
function resultToCsv() {
  console.log('Number of records to be written to csv: ' + final_result.length); // Why is this Zero ?? Expected an array of objects here
  csvWriter
    .writeRecords(final_result)
    .then(() => console.log('The CSV file was written successfully'));
}

//Function calls
run();
resultToCsv();

@jared, thanks for your comment but that situation does not apply to this — Nick, Mar 31 '20 at 21:15
@prov, my problem is right at the end when I get to writing the results to csv, not obtaining the data. But correct me if I'm wrong. — Nick, Mar 31 '20 at 21:18
@Nick — Yes it does. You call `run` which triggers a bunch of asynchronous stuff, then (before the asynchronous stuff has finished) you call `resultToCsv` while `final_result` is still an empty array. — Quentin, Mar 31 '20 at 21:26
That was exactly my thought, why it comes to this problem. You have to wait that the run function has finished before you are calling resultToCsv — ProV, Apr 01 '20 at 08:27

score 0 · Answer 1 · answered Mar 31 '20 at 21:24

This issue here is that when you do this:

run();
resultToCsv();

Those two functions are being called immediately. Because run() is using asynchronous methods, it won't have finished reading the stream before resultToCsv() is called. So you want to wait for the stream to be done being read before calling that.

function run() {
  fs.createReadStream('yodeldata.csv')
    .on('error', () => console.log('error'))
    .pipe(csv())
    .on('data', row => {
      output(row); //calling function 2 of 3
    })
    .on('close', resultToCsv);

}

Cannot access global array and write it to csv

1 Answers1