1

So I am trying to do some client side validation on an app I am working on, and I want to be able to check the number of columns in a CSV that the user uploads. The CSVs could potentially be very large (up to 30 thousand rows) so I'm worried that my current method, which reads the entire file, is not the best way to do it.

This is what I have right now. 

// Function to validate the File Upload
$(document).ready(function() {
    $("#csv").change(function () {
        console.log("Read CSV Function Triggered");
        let theFile = document.getElementById("csv").files[0];
        let reader = new FileReader();
        reader.addEventListener('load', readFile);
        reader.readAsText(theFile);

        function readFile(event) {
            let csvText = event.target.result;
            let firstLine = csvText.split('\n').shift(); // First Line
            $("#result").html(firstLine);
        }
    });
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>

<input type="file" id="csv" />

<br>

<span id="result">0</span>

In the example I just make the first line display on the page, but in the app I have a parser library that I couldn't include in this example which converts the first line to an array and then I get the length of the array. So like I said before, this works, but I am worried that when large CSVs are uploaded it may affect browser performance.

Any suggestions?

pythonweb
  • 1,024
  • 2
  • 11
  • 26
  • see this post : https://stackoverflow.com/questions/2952805/count-number-of-lines-in-csv-with-javascript – MajiD Jul 11 '18 at 23:34
  • Just load the CVS file row for row, checking colcount as you go, until you hit a row that does not have the required column count, and then stop, going "Error on row X, incorrect number of columns" and then let the user decide whether to fix that problem themselves or not. Auto-generated CSV files will never run into this problem, so if someone creates a manual CSV, telling them to manually fix it seems more than fair. – Mike 'Pomax' Kamermans Jul 11 '18 at 23:36
  • 1
    If you use library papaparse.js it has chunking and web worker options. Chunking can cut down memory needed and web worker offloads the task from active page – charlietfl Jul 11 '18 at 23:39
  • @charlietfl So are the web workers built into the functions in papa-parse or would I need to set them up manually? – pythonweb Jul 11 '18 at 23:46
  • 1
    Web workers are built into browser https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API . As for the library it has numerous configuration options – charlietfl Jul 11 '18 at 23:50

0 Answers0