Reading csv with fast-csv in node.js when 3 columns have same captions

Question

I am trying to read a CSV file with fast-csv in Node.js. The csv file has a header row which contains 90 columns. 87 of them have unique names. But 3 columns have the same header name "Transformation". This is obviously a semantic mistake. Fast-csv is complaining that headers are not unique and does not read the CSV. I know, all column names should be different, but this is how I get the CSV file. Is there a way to read the headers and if some column has a name that another column has, the name automatically converts to "Transformation_int", with int being some random number?

csv.parseFile(csvFileName,
       { headers: true, delimiter: ";", 
         headers: headers => headers.map(
          h => {h.replace( /[\r\n]+/gm, " " )}) 
        })
    .on ("data", function (row) { <doing something> })
    .on ("error", error => {console.log("This error occured:",error)})
    .on ("end", () => {      console.log("Import finished")});

score 2 · Accepted Answer · answered Jun 24 '22 at 20:29

you could not use headers at all, and then the first row will contain headers, and you can manually process the rest of the data with reference to the first row as you need, and even keep duplicate names:

headers: false

or, you can pass a transform function which detects duplicates, and also renames them

here's a quick example which renames duplicate headers by adding random number:

csv.parseFile(csvFileName, {
        delimiter: ";",
        headers: headers => {

            // rename duplicate
            headers.map((el, i, ar) => {
                if (ar.indexOf(el) !== i) {
                    headers[i] = `${el}_${i}`;
                }
            });

            return headers;
        }
    })
    .on("data", function(row) {
        console.log('row', row);
    })
    .on("error", error => {
        console.log("This error occured:", error)
    })
    .on("end", () => {
        console.log("Import finished")
    });

Thanks @traynor. That was exactly what I was looking for. For some reason I thought that at the point where you apply the transformation function the headers are already read out, thus they must be well formed in the first place. — A.T., Jun 25 '22 at 07:06

Reading csv with fast-csv in node.js when 3 columns have same captions

1 Answers1