7

I am trying to send a csv file which is uploaded by the user, from browser to nodejs server for processing (file is over 50 mb, so the page becomes unresponsive). I'm using XMLHttpRequest for this purpose. I cannot find a solution to this. Any help is appreciated.

Javascript code

 var csv = document.getElementById('inputFile').files[0];
 var request = new XMLHttpRequest();
 request.open("POST", "/handleFile", true);
 request.setRequestHeader("Content-type", "text/csv");
 request.onreadystatechange = function() {
   if (request.readyState === XMLHttpRequest.DONE && request.status === 200) {
     console.log("yey");
   }
 }

 request.send(csv);

NodeJS server

 var express = require('express')
 var app = express()
 var bodyparser = require('body-parser')

 app.post('/handleFile', function(req, res) {
   console.log(req.body); // getting {} empty object here....
   console.log(req);

   var csv = req.body;
   var lines = csv.split("\n");
   var result = [];
   var headers = lines[0].split("\t");

   for (var i = 1; i < lines.length; i++) {
     var obj = {};
     var currentline = lines[i].split("\t");

     for (var j = 0; j < headers.length; j++) {
       obj[headers[j]] = currentline[j];
     }

     result.push(obj);
   }

   fileData = result;
 });

What did I do wrong? Is the XMLHttpRequest used incorrectly? or there is some other thing that i did not understand ? why is there no data in req.body even though its a post request. Or is there any other way to send a csv/text file to nodejs server from front end.

This question is not a duplicate because, the body-parser i.e. the middleware responsible for parsing the req.body does not handle text/csv and multipart/form-data . The above link is not the correct solution.

Mr. Polywhirl
  • 42,981
  • 12
  • 84
  • 132
Subhash
  • 363
  • 1
  • 4
  • 15
  • Slice the file into chunks and send them one by one: http://stackoverflow.com/questions/20212851/slice-large-file-into-chunks-and-upload-using-ajax-and-html5-filereader – Emil S. Jørgensen Feb 07 '17 at 10:26
  • 1
    i think you should have a look at [multer](https://github.com/expressjs/multer) – Sudheesh Singanamalla Feb 07 '17 at 10:26
  • Or [resumablejs](http://resumablejs.com/). – Emil S. Jørgensen Feb 07 '17 at 10:27
  • the issue is not the size, why am i not getting anything in my nodejs server? what did i do wrong? – Subhash Feb 07 '17 at 10:29
  • @Subhash — That link is described as "Node.js middleware for handling `multipart/form-data`". How would that help since the OP is sending `text/cvs` data and not `multipart/form-data`? – Quentin Feb 07 '17 at 10:34
  • @EmilS.Jørgensen — That's a client library. How would it help parse the data on the server? – Quentin Feb 07 '17 at 10:35
  • @Quentin It won't help the server parse the data, it will get the data to the server without making the page go unresponsive. Once the file is fully uploaded it can be handled asynchronously server side. This question is about `is there any other way to send a csv/text file to nodejs server from front end` not how to parse it once it's there. – Emil S. Jørgensen Feb 07 '17 at 10:38
  • @EmilS.Jørgensen it wont matter if i slice it into smaller part or not, i am not receiving anything at server side. Right now i have a small demo file which shouldn't make the page unresponsive. – Subhash Feb 07 '17 at 10:45
  • @Quentin looking in to the link you gave, it might be possible that the body-parser is unable to parse the csv file, ill try changing the options. I;ll update you guys as soon as I find anything. – Subhash Feb 07 '17 at 10:46
  • @Subhash — Don't "change the options", use the raw body option described in the accepted answer on the duplicate question. – Quentin Feb 07 '17 at 10:47
  • @SudheeshSinganamalla i tried multer and it worked. But it converts the file into buffer. Any module that you use to convert it back to csv or text file? – Subhash Feb 07 '17 at 11:28
  • @Quentin I tried the accepted answer on duplicate question, but the result was the same. Anyways will post an answer once i get it all done. – Subhash Feb 07 '17 at 11:29
  • @Subhash You can use the [toString()](https://github.com/expressjs/multer/issues/290) that's available as described here to convert it to string, but I think once you do that, you should split it by `\n` which will give you an array and then merge them back into a file by looping through the list. – Sudheesh Singanamalla Feb 07 '17 at 16:42
  • @SudheeshSinganamalla i did use toString(). see the answer below, i wanted to do some operations so didn't need to split then merge again. Anyways ty. – Subhash Feb 08 '17 at 07:13
  • Glad to hear that it worked. – Sudheesh Singanamalla Feb 08 '17 at 07:15

1 Answers1

7

So, after looking around, I found that the problem was not my XMLHttpRequest. The request was received by the server just fine, but the body-parser could not parse the text/csv and multipart/form-data content-type. Here is the step by step answer to this problem.

  1. In the client/browser-end whenever you are sending a large file to the server, convert it into multipart/form-data . It is the correct way of sending a text/csv/anyfile to the server.

    var csv=document.getElementById('inputFile').files[0];
var formData=new FormData();
formData.append("uploadCsv",csv);
var request = new XMLHttpRequest();

 //here you can set the request header to set the content type, this can be avoided.
 //The browser sets the setRequestHeader and other headers by default based on the formData that is being passed in the request.
 request.setRequestHeader("Content-type", "multipart/form-data"); //----(*)
 request.open("POST","/handleFile", true);
request.onreadystatechange = function (){
    if(request.readyState === XMLHttpRequest.DONE && request.status === 200) {
    console.log("yey");
    }
}

request.send(formData);

So, this is how you'll send your http request to the nodejs server.

  1. On Node js server: Normally for application/json or any other request type the body-parser would have worked fine. But for large data and files i.e. multipart/form-data body-parser cannot parse the req.body. Thus it will give req.body as {} (empty object). Read about body-parser here.

So for these content-type you can use other middleware for handleling the request. Some are multer,multiparty,busboy etc. I used multer. Here is the code snipet.

    //EXPRESS
    var express = require('express')
    var app = express()

    var config=require('./config.js');
    //multer
    var multer  = require('multer');
    var upload = multer();
    app.post('/handleFile',upload.single('uploadCsv'), function(req, res, next) {
          // req.file is the `uploadCsv` file 
          // req.body will hold the text fields, if there were any 
          console.log(req.file);
          // the buffer here containes your file data in a byte array 
          var csv=req.file.buffer.toString('utf8');
     });

NOTE: This will still give you an error in nodejs server. hint: It has something to do with the line (*). Try removing it and see what happens. Google the rest ;)

Subhash
  • 363
  • 1
  • 4
  • 15