How to parse a CSV file on Node/Express from an Axios GET request?

Question

I'm doing a GET request towards a URL-endpoint that sends back a CSV-file on a Node/Express setup using axios.

How do I read this file and parse it into an array with objects for each row of data? I like the examples of how "neat-csv" handles parsing but I'm open to suggestions.

const response = await axios.get("url/end/point",{ responseType: 'blob',}); 

I have no real examples of implementations so far other than the request since I have no clear view on where to begin and I can't find any good guides on the net. All guides I find involves requesting a file and then saving it to the filesystem which i don't want to. I simply want to request the file, parse it and then broadcast the data in the CSV-file through a Socket.io connection I have.

Loggin response.data clearly gives a data output but not formatted of course. Just raw data/text.

In the response there's a Content-Disposition parameter that reads attachment; filename=name_of_file.csv

If that helps?

Cheers and thanks in advance!

Answer 1

Probably you've already found the answer, but i think it's worthy to add something here, just in case:

1) Fetch the CSV from an URL

So you're getting the CSV as a BLOB from an axios request. So far, so good! As you stated:

const response = await axios.get("url/end/point",{ responseType: 'blob',});
const file = response.data;

Here, you have (and find online) different ways to make an object out of it.

2) Get value as a string

First, to log it as a proper string, you can do:

file.text().then((csvStr) => {
  console.log(csvStr);
})

It will log something like this (as a string):

Title1,Title2,Title3
one,two,three
example1,example2,example3

3) Parse it to an object

Here, you have different options. Mainly:

A) Write your own function to parse the string as a CSV into an object

One good example I found here at StackOverflow is from this answer from Wesley Smith

function csvJSON(csvStr){
  var lines=csvStr.split("\n");
  var result = [];

  // NOTE: If your columns contain commas in their values, you'll need
  // to deal with those before doing the next step 
  // (you might convert them to &&& or something, then covert them back later)
  // jsfiddle showing the issue https://jsfiddle.net/
  var headers=lines[0].split(",");

  for(var i=1;i<lines.length;i++){

      var obj = {};
      var currentline=lines[i].split(",");

      for(var j=0;j<headers.length;j++){
          obj[headers[j]] = currentline[j];
      }

      result.push(obj);

  }
  return result; //JavaScript object
}

Now, if you do:

const jsonObj = csvJSON(csvStr);
console.log(jsonObj);

You will get inside jsonObj the CSV in the form of the object you wanted!

B) Use some third-party library that does that for you

Here, you might find some options. Feel free to pick whatever you prefer. I picked CSVTOJSON and will show it following the same example:

import csvToJson from 'csvtojson';

...

csvToJson()
  .fromString(csvStr)
  .then((jsonObj)=>{
    console.log(jsonObj);
  });

Like this, you will also get inside jsonObj the CSV in the form of the object you wanted!

With any option A or B, you will get:

NOTE: In both options, you might have to implement the conversion inside the then() block as it's there when you have the string value ready to be used:

... 
file.text().then((csvStr) => {
  // Option A
  const jsonObj1 = csvJSON(csvStr);
  console.log(jsonObj1);

  // Option B
  csvToJson()
    .fromString(csvStr)
    .then((jsonObj2)=>{
      console.log(jsonObj2);
     })
    });

What option to pick?

That depends on you and the CSV file you're using. It's good to consider:

• If you have a well-known and small file, maybe using a third-party library is overkilling.
• On the other side, implementing your own function when your file may have some tricky content (like inner commas, some other separators or extra characters that might generate a fail conversion), then function might require extra work and, therefore, headaches.
• Some other thing you may want to consider is the size of your file. If it's extremely large, and you're implementing your own function, you will want to do it in a very performant way. If you're using a third-party library (a good one), you won't have to worry too much about that.

Answer 2

If you want to basically get a CSV string sent from the backend and parse it into an array to manipulate, its possible. I would personally do all the formatting on the backend though so it returns what you need (if this is possible).

You said you wanted to parse the string into an array of objects? Seems unclear to me but if you let me know what you mean I can help you. The below code would parse the file into an array, where each position in the array represents a line of the CSV.

const response = await axios.get("url/end/point",{ responseType: 'blob',}); 
const CSV = response.data
// CSV === 'test,this,idea\nIs,it,working\nYes,it,is'
const lineArray = CSV.split('\n')
// lineArray === ['test,this,idea', 'Is,it,working', 'Yes,it,is']