Sails.js Skipper: How to read the uploaded file stream during upload?

Question

I am using sails version 0.11 which comes bundled with skipper for file uploads. I need to upload big files of over 100 MB in size at least.

Now I don't want the file upload to complete before I start further processing of the file on server. I want to read the file stream while it is being uploaded. Here is how I am trying to achieve the same:

var csv = require("fast-csv");

bulk: function (req, res){

  function parseEntry(entry){
    return entry
  } 

  var inputStream = req.file('employees');
  var csvStream = csv();
  inputStream.pipe(csvStream); 

  csvStream.on("data", function (data){
    count++
    // skip invalid records
    var parsedData = parseEntry(data);
    console.log(parsedData);
  });

  csvStream.on("end", function(){
    // console.log("total time taken in minutes:", timeTaken);
    return res.json(200, {"status": 1, "message": "successfully uploaded the file"});
  });

}

But my logs only show up the end event and no data event is being captured. I read in the documentation that req.file("filename") will return stream of file streams. But how do I access the particular file stream I need since I am only uploading a single file?

Probably because Skipper users streams 2, while your cvsStream using old style stream — securecurve, Dec 29 '15 at 10:18
I have the same exact problem, did you manage parsing your csv as a stream? — hammady, Feb 25 '16 at 11:33
nope. We did not reach the situation where file size was over a few MBs. If I return to this problem and solve it someday, I will update this thread for sure — Mandeep Singh, Feb 25 '16 at 13:17
finally did it, just wrote skipper-csv module, please keep an eye on http://github.com/hammady, i will push it shortly! — hammady, Feb 25 '16 at 14:05

score 3 · Accepted Answer · edited Sep 26 '16 at 21:20

Just published skipper-csv ;)

npm install skipper-csv --save

Use it the same way you would use other adapters. It uses csv-parse to do the actual parsing. You typically pass csv-parse options to the upload function. Additionally you pass the rowHandler function that is called for each parsed row. The fd (file descriptor) is also passed in case you are uploading multiple files at the same time. Here is an example:

req.file('files').upload({
  adapter: require('skipper-csv'),
  csvOptions: {delimiter: ',', columns: true},
  rowHandler: function(row, fd){
    console.log(fd, row);
  }
}, function (err, files) {
  if (err)
    return res.serverError(err);

  return res.json({
    message: "Uploaded " + files.length + " CSV files!",
    files: files
  });

});

Sails.js Skipper: How to read the uploaded file stream during upload?

1 Answers1

Linked