How do I read a CSV file into an array using d3js v5

Question

I am creating a data dashboard that should allow the user to point to a CSV file on their file system, which is then uploaded into the dashboard to perform all kinds of visualisation actions on it.

I am using a nice d3js example for the visualisation, but none of the examples I found include reading the CSV file and I cannot make it work. I can read the CSV file and log the contents to the console, but whatever I do to pass the data into an array (which I can subsequently use as the basic data to visualise), there is simply no way my array becomes available as a JS or even a d3js variable.

Here is part of the file I am using - giving numbers of cars that are in use in various EU countries.

Country;Car;Van;Truck;Total
Austria;4748048;375163;78539;5201750
Belgium;5587415;678801;159623;6425839
Croatia;1489338;127395;45757;1662490

As the delimiter is a semicolon, I am using the dsv function so that I can explicitly set the delimiter.

function getCSVdata( ) {
    var myData = [];
    d3.dsv(";","EU-cars.csv", function(d) {
        return {
            State : d.Country,
            freq: { 
                low  : d.Car,
                med  : d.Van,
                high : d.Truck
            }
        };
    }).then( function(d) {
        myData.push( d );
        console.log( d );
    }); 
    return myData;
}

The return value is assigned to a variable named 'freqData' (just using the existing sample code here to try and localise the problem - the used code works file if I declare the object in JS instead of reading it from a CSV file).

Writing the 'freqData' object to my console shows that it is an array with 1 element. Writing 'freqData[0]' to the console shows an array with all expected 26 records. But trying to write 'freqData[0].length' gives an error as that is undefined. Trying to use alert causes an error.

This may be something simple for those who have tons of d3js and other JS experience but it turns out to be too complex for newbies.

I need to use the latest (v5) of d3js, which is using the Promise object that is totally new to me. I cannot figure it out by reading tons of partial info on all kinds of supposedly known methods. If someone can just post a fully working example of reading a CSV and passing the data to a JS object that would be wonderful. I am running out of steam and time.

Thanks

4everJang

Answer 1

When you run your code you should see that you are indeed loading the data. When I use your example data in a dsv/csv file and run your function I get the following logged to console:

So it appears as both your load of the dsv file is successful, and your row function is successful, based on the console log in the then method.

Let's take a closer look at the .then method, this runs after the file has loaded and the row function been applied - or, once the promise has been fulfilled. So the key part here is the then method doesn't execute the provided function until after the promise is fulfilled. While we are waiting for the the promise to be fulfilled, code continues to be executed, in your case the return statement executes - before the data is fully loaded and therefore before the then method executes. This means you aren't returning any data, and this is your problem.

The simplest and probably most common d3 patterns to fetch the data with your row function is to either place code that requires the loaded data in the .then method's fulfillment function:

function getCSVdata( ) {
    d3.dsv(";","https://raw.githubusercontent.com/Andrew-Reid/SO/master/dsv.csv", function(d) {
        return {
            State : d.Country,
            freq: { 
                low  : d.Car,
                med  : d.Van,
                high : d.Truck
            }
        };
    }).then(function(data) {
      console.log("file has " + data.length + " rows");
      logger(data);
    }); 
}

getCSVdata();

function logger(data) {
 console.log(data);
}

.as-console-wrapper { max-height: 100% !important; top: 0; }

<script src="https://d3js.org/d3.v5.min.js"></script>

Or you could take a slightly different form:

function getCSVdata( ) {
    d3.dsv(";","https://raw.githubusercontent.com/Andrew-Reid/SO/master/dsv.csv", function(d) {
        return {
            State : d.Country,
            freq: { 
                low  : d.Car,
                med  : d.Van,
                high : d.Truck
            }
        };
    }).then(fulfilled); 
}

getCSVdata();

function fulfilled(data) {
 // do stuff here:
    console.log(data);
}

.as-console-wrapper { max-height: 100% !important; top: 0; }

<script src="https://d3js.org/d3.v5.min.js"></script>