Race condition Node JS

Question

I have been going through documentation but for whatever reason I cannot see what is happening here.

I have two functions, 1 downloads a csv file via a url and the next function takes that csv file and converts it to JSON

FileDownload.js

const http = require('http');
const fs = require('fs');

module.exports = function(url, dest){
  var file = fs.createWriteStream(dest);
  return new Promise((resolve, reject) => {
  var responseSent = false; // flag to make sure that response is sent only once.
  http.get(url, response => {
    response.pipe(file);
      file.on('finish', () =>{
        file.close(() => {
          if(responseSent)  return;
            responseSent = true;
            resolve();
          });
       });
     }).on('error', err => {
       if(responseSent)  return;
         responseSent = true;
         reject(err);
       });
     });
}

CSVToJson.js

const csvjson = require('csvjson');
const fs = require('fs');
const write_file = require('../helpers/WriteToFile');

function csvToJson(csv_file, json_path) {

  var data = fs.readFileSync(csv_file, { encoding : 'utf8'});
  var options = {
    delimiter : ',',
    quote     : '"'
  };
  const json_data = csvjson.toObject(data, options);
  write_file(json_path, json_data)
}

module.exports = {
 csvToJson: csvToJson
}

I then call these functions in succession, but CSVToJson finishes before FileDownload, thus causing a failure (no data to convert)

data.js

const premier_league = require('./scripts/getCSVData')
const csv_to_json = require('./scripts/csvToJson')

const pl_json_path = './team_data/premier_league/clubs/all/premier_league.json'
const pl_csv_path = './csv/premier_league/premier_league.csv'

// Get Premier League CSV file
premier_league.getPremierLeagueData();

// Convert CSV Data to json
csv_to_json.csvToJson(pl_csv_path, pl_json_path )

What am I doing wrong at the file download stage? I haven't quite got my head around callbacks and I only want to execute the csv conversion once the file has been downloaded.

is it not just call the `csvToJson` function when the promise has resolved? — Paul Fitzgerald, Nov 14 '17 at 09:16

Sten Muchow · Answer 1 · 2017-11-14T13:48:01.863

You are using Node so the integration of Promises to handle these async issue is pretty easy.

You can then basically do then FileDownLoad.then(CSVToJson)

99% of JS issues seem to boil down to async problems :D

Having looked further into the code I see you have this line as well.

fs.readFileSync(csv_file, { encoding : 'utf8'});

This is another async issue - you cant be sure this is done before you call csvjson method. Again need promises.

======= Update ==========

function csvToJson(csvFile, jsonPath) {
    fs.readFile(csvFile, 'utf8', (err, data) => {
        if(err) //handle me

        const options = {
            delimiter : ',',
            quote     : '"'
          };
        const jsonData = csvjson.toObject(data, options);
        write_file(jsonPath, jsonData)
    });
}

I guess something like the above. You could wrap in some promises and make the api a nice promised based one which is always nice ;)

thank you for your answer, I dont suppose you are able to assist further with some code example could you please? not necessarily answering my question , but something to refer to so i can learn ? thanks — Richlewis, Nov 14 '17 at 13:07

Race condition Node JS

1 Answers1