Node.js: fs writestream stops writing to file when the file gets too big

Question

I'm scraping a webpage with lots of data on it, formatted as an HTML table. You have to submit a form to generate the table. My node script submits all the permutations of the form, and each time scrapes the resulting table, turning each row into a line of data.

The problem is, when I write the data to some file, it stops working when the file gets to be about 10MB in size. Sometimes it's a little less; sometimes a little more. I have tried writing the file as .csv, .json, and .txt, and each time the same problem occurs.

I am using fs to perform this task. The relevant code is:

var fs = require("fs");
var stream = fs.createWriteStream("data.csv"); // can also be .json or .txt

stream.write(line_of_data);

I can console.log(line_of_data) and it works fine, all the way through until there's no data left to scrape. But at about 10MB, the output file won't accept any more lines of data. The stopping point seems almost completely arbitrary -- every time I run the script, it stops writing at a different point. I have plently of storage space on my hard drive, so the problem must have to do with something else.

Can you store different resulting tables in different files and in the end merge them with something other than nodeJS ? — atakanyenel, Jan 05 '17 at 13:00
see this answer http://stackoverflow.com/questions/9486683/writing-large-files-with-node-js — Shubham Batra, Jan 05 '17 at 13:04

score 0 · Answer 1 · answered Jan 09 '17 at 06:56

I ended up using MongoDB to store the data. To install MongoDB as a node module, run npm install mongodb --save. The relevant javascript is:

MongoClient = require("mongodb").MongoClient;
MongoClient.connect("mongodb://localhost:27017/database", function(err, db) {

  if (!err) {

    // set up mongodb collection
    db.createCollection("collection", function(err, collection) {}); 
    var collection = db.collection("collection");

    // after scraping data... 
    // insert a data object (line_of_code)
    collection.insert(line_of_data, {w: 1}, function(err, result) {
      if (err) console.log(err);
    });

  }

});

Some commands to convert the data:

Export as CSV: mongoexport --db database --collection collection --out data.csv --type=csv --fields 'field1, field2, field3, etc.'
Convert to JSON: csvtojson data.csv > data.json (requires csvtojson)
Validate JSON: jsonlint data.json (requires jsonlint)

Node.js: fs writestream stops writing to file when the file gets too big

1 Answers1