How to read files .csv very big with FileReader API javascript, html/jquery only

Question

I'm trying to read a file .csv over 40 MB (with more 20.000 lines), viewing it as a table in html. The system I am designing will be in pure HTML + JQuery only. My worksheet .csv format that is:

=======================================================================
| rank ||  place           || population ||    lat    ||      lon     |
=======================================================================
|  1   || New York city    ||  8175133   || 40.71455  ||  -74.007124  |
-----------------------------------------------------------------------
|  2   || Los Angeles city ||  3792621   || 34.05349  ||  -118.245323 |
-----------------------------------------------------------------------
..........................Thousands of lines..........................

I have a textbox filter that makes me just showing the line that has the same number entered in the field. I have a textbox filter that makes me just showing the line that has the same number entered in the field. And then uploading the file it breaks the browser. :( Follow my code:

var fileInput = document.getElementById('fileInput');
var fileDisplayArea = document.getElementById('fileDisplayArea');

        fileInput.addEventListener('change', function(e) {
            var file = fileInput.files[0];

                var reader = new FileReader();

                reader.onload = function(e) {

                    //Convert .csv to table html
                    var data = reader.result; 
                    var rank = $("#rank").val(); 

                    if(rank == ""){
                        alert("digit rank number");
                        fileInput.value = "";
                     }
                     else{



                    data = data.replace(/\s/, " #"); 
                    data = data.replace(/([0-9]\s)/g, "$1#"); 
                    var lines = data.split("#"), 
                        output = [],
                        i;
                        for (i = 0; i < lines.length; i++)
                        output.push("<tr><td>" + lines[i].slice(0,-1).split(",").join("</td><td>") + "</td></tr>");
                        output = "<table>" + output.join("") + "</table>";

var valuefinal = $(output).find('tr').filter(function(){ 
    return $(this).children('td').eq(0).text() == rank;
});

 $('#fileDisplayArea').append(valuefinal);

                     }
                }

                reader.readAsBinaryString(file);    


        });

Is it possible to do something that I optimize my code and not let my browser and break my file to be read? I tried to split the file into pieces and then seal it. (BLOB) But this way he kept crashing my browser too. :(

(Excuse me if something was not clear and enlightening about my question to you, my english is bad.) DEMO CODE JSFiddle

My recommendation would be to design a much better UI that doesn't force the user to deal with a 20,000 line table. That's just not going to be a good UI even if you do succeed in reading the whole thing. — jfriend00, Jun 22 '14 at 23:12
@Derek朕會功夫 Pay attention to this part of my question: "I tried to split the file into pieces and then seal it. (BLOB) But this way he kept crashing my browser too. :( " — user3743128, Jun 22 '14 at 23:27
@jfriend00 In the table with 20 000 lines served to consult with another script I created. The User will not have to deal with 20,000 lines — user3743128, Jun 22 '14 at 23:28
If the user doesn't have to see 20,000 lines, then why are you putting all 20,000 lines of the data into a ``. That's the slowest form of storage you can make. You can probably make the browser not crash (in desktop browsers only) by breaking your processing into chunks where you process each chunk on a `setTimeout(fn, 1)`, but it's still a completely flawed design. Better to fix the design than trying to just get a bigger hammer. — jfriend00, Jun 22 '14 at 23:57
@jfriend00 I need to do a specific search in accordance with the User ID number to enter, I see no other alternative than this to convert the file `.csv` in `tables` for doing this research. Is there any other way I can do this research without passing this file `.csv` to html. Make a direct search in the file? — user3743128, Jun 23 '14 at 00:14
You can just process the data into a javascript array of objects. No need to put it in HTML just to search it. — jfriend00, Jun 23 '14 at 02:05
@jfriend00 And how can I apply this in my code? is there any example of this? — user3743128, Jun 23 '14 at 11:59
Getting back to the issue though; is it the upload that crashes or generating a table that size? There is (theoretically) no limit on the size of the file that FileReader excepts. It's hard to test without uploading a file... — Jorg, Jun 23 '14 at 12:04
@jfriend00 The browser stops responding, crashes and closes. I believe it is due to loop it performs :( — user3743128, Jun 23 '14 at 16:43
If the browser is complaining about being a script being non-responsive, then you need to chunk your work using `setTimeout()` as shown in this example: http://stackoverflow.com/questions/10344498/best-way-to-iterate-over-an-array-without-blocking-the-ui/10344560#10344560 or use webworkers to load the CSV into a JS data structure in the background. You probably also should stop storing the data in a `` just for searching purposes too because putting your data into just a JS data structure would likely be a ton faster. — jfriend00, Jun 23 '14 at 16:46
@jfriend00 How can I implement this code in your example of my code? And as you would for my other javascript seek a specific information in this table form? — user3743128, Jun 23 '14 at 17:00
@jfriend00 I've tried to break in pieces the file and then seal it but time to put it on the table as html browser again break. Is there any method that I can perform a query on the worksheet ´.csv´ without playing it in html? — user3743128, Jun 23 '14 at 17:13
I was asking what part of my suggestions do you not understand? I've given you a direction to go and all you've said is how do I implement this? I want to know what part of implementing it do you not understand? The general idea is that you open the file, set up a few state variables, read and process N bytes of the file, then do a `setTimeout()` to let the browser breathe, then read and process N more bytes, repeat until done as is shown in the linked post. To query the file directly, you'd have to put it into a database (probably with a server behind it) that had that capability. — jfriend00, Jun 23 '14 at 17:16

score 3 · Answer 1 · answered Apr 15 '16 at 15:55

3

As I found this thread on my research to parse a very big CSV file in the parser, I wanted to add that my final solution was to use Papa Parse:

http://papaparse.com/

answered Apr 15 '16 at 15:55

Nukey

326
3
10

How to read files .csv very big with FileReader API javascript, html/jquery only

1 Answers1