Javascript using File.Reader() to read line by line

Question

This question is close but not quite close enough.

My HTML5 application reads a CSV file (although it applies to text as well) and displays some of the data on screen.

The problem I have is that the CSV files can be huge (with a 1GB file size limit). The good news is, I only need to display some of the data from the CSV file at any point.

The idea is something like (psudeo code)

var content;
var reader =  OpenReader(myCsvFile)
var line = 0;

while (reader.hasLinesRemaning)
    if (line % 10 == 1)
      content = currentLine;
Loop to next line

There are enough articles about how to read the CSV file, I'm using

function openCSVFile(csvFileName){
    var r = new FileReader();
    r.onload = function(e) {
        var contents = e.target.result;
        var s = "";
    };  
    r.readAsText(csvFileName);
}

but, I can't see how to read line at a time in Javascript OR even if it's possible.

My CSV data looks like

Some detail: date, ,
More detail: time, ,
val1, val2
val11, val12
#val11, val12
val21, val22

I need to strip out the first 2 lines, and also consider what to do with the line starting with a # (hence why I need to read through line at a time)

So, other than loading the lot into memory, do I have any options to read line at a time?

http://papaparse.com/docs may help, use preview option – Christophe Roussy Mar 23 '17 at 13:34 — Christophe Roussy, Mar 23 '17 at 13:34

score 9 · Accepted Answer · edited May 23 '17 at 11:46

There is no readLine() method to do this as of now. However, some ideas to explore:

Reading from a blob does fire progress events. While it is not required by the specification, the engine might prematurely populate the .result property similar to an XMLHttpRequest.
The Streams API drafts a streaming .read(size) method for file readers. I don't think it is already implemented anywhere, though.
Blobs do have a slice method which returns a new Blob containing a part of the original data. The spec and the synchronous nature of the operation suggest that this is done via references, not copying, and should be quite performant. This would allow you to read the huge file chunk-by-chunk.

Admittedly, none of these methods do automatically stop at line endings. You will need to buffer the chunks manually, break them into lines and shift them out once they are complete. Also, these operations are working on bytes, not on characters, so there might be encoding problems with multi-byte characters that need to be handled.

See also: Reading line-by-line file in JavaScript on client side

Javascript using File.Reader() to read line by line

1 Answers1

Linked