How do I read a file line by line in Vanilla JavaScript?

Question

I have a file that i want to read(words.txt). I would like to save it in an array, but it is too long to include in the file, so I put it in a separate file. I can use a URL to get it, but I also have the file in the same folder as the javascript file. What would be the simplest way to do this without using jQuery or Node.js?

Answer 1

It seems like you are trying to read a very large file in chunks, so as not to crash the browser with too much memory, am I right? If not, let me know.

While in general, to read it "line by line", people (How to use Javascript to read local text file and read line by line?) usually just load the entire file, then split it by "\n", but that fails to address the issue of crashing the browser with an overtly large file.

What you're looking for is the progress event for the FileReader, or XMLHttpRequest classes, depending on whether you are fetching local, user-selected files or files from a server.

It should allow you to read parts of a larger file in chunks. As the chunks come in, you can keep track of the new line characters (\n), and keep each set of text in a new array, accordingly.

For keeping track of the chunks, XMLHttpRequest chunked response, only read last response in progress, if you're fine with keeping an accumulated amount of response data in memory, if not you can try the array-buffer approach mentioned at How to write javascript in client side to receive and parse `chunked` response in time?, or even better, to use the ReadableStream class as mentioned in an answer there, together with fetch.

Based on that answer as well as others, I've attempted to put together a function which will give you a callback each line is read, and read the total URL by chunks, even though the chunks themselves may be slightly more than 1 line at a time, but it's the closest I could think of with client-side JavaScript, let me know if it works:

function readLineByLine(url, callback) {
    fetch(url)
    .then(response => {
        let fetchReader = response.body.getReader(),
            decoder = new TextDecoder(),
            currentLineData = "",
            totalLines = [];
        fetchReader.read()
        .then(function readData(progress) {

            let thisJustInSomeData = decoder.decode(
                progress.value || "",
                {
                    stream: !progress.done
                }
            ),
                lines = thisJustInSomeData.split("\n");

            if(lines.length < 2) {
                currentLineData += lines[0];
            } else {
                lines.forEach((x, i, a) => {
                    currentLineData += x;
                    totalLines.push(currentLineData);

                    if(i < lines.length - 1) {
                        currentLineData = "";
                    }
                });
            }

            totalLines.forEach(line => {
                callback({line});
            });

            totalLines = []
            if(progress.done) {
                callback({done: progress.done})
                return;
            }

            return readData();
        })
    });
}

then to call it

readLineByLine("someURLorTxtFileWithLotsOfLines", line => console.log(line));