1

I am looking to use swagger to upload a csv file and then process the file in my nodejs app, by looping through each row. The swagger part is fairly straight forward, but I can't read the file to be able to process the data.

I have used the basic swagger example below that works fine to serve the file, but I am unsure how to handle it in the backend.

paths:
  /upload:
    post:
      summary: "Uploads a file."
      operationId: "uploadPOST"
      consumes:
      - "multipart/form-data"
      parameters:
      - name: "upfile"
        in: "formData"
        description: "The file to upload."
        required: false
        type: "file"
      responses:
        201:
          description: "An object with user details"
      x-swagger-router-controller: "Default"

I then have a very basic function in the backend:

const fs = require('fs'); 
const csv = require('csv-parser');

module.exports.uploadPOST = function uploadPOST (req, res, next) {

  var fileValue = req.swagger.params['upfile'].value;
  console.log(fileValue);

  fs.createReadStream(fileValue)
  .pipe(csv())
  .on('data', function(data){
    console.log(data);
  })
  .on('end',function(){
      console.log("Done")
  });  

};

When I log the file I receive:

{ fieldname: 'upfile',
  originalname: 'test.csv',
  encoding: '7bit',
  mimetype: 'application/octet-stream',
  buffer:
   <Buffer 63 6f 6c 75 6d 6e 5f 61 2c 63 6f 6c 75 6d 6e 5f 62 2c 63 6f 6c 75 6d 6e 5f 63 2c 63 6f 6c 75 6d 6e 5f 64 2c 63 6f 6c 75 6d 6e 5f 65 2c 63 6f 6c 75 6d ... >,
  size: 251 }

Now when I try to read the stream from "req.swagger.params['upfile'].value", I receive the following error:

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of type string, Buffer, or URL

So I thought maybe I can then pass in the buffer value via: "req.swagger.params['upfile'].value.buffer" but then I get the error:

Error: ENOENT: no such file or directory, open 'column_a,column_b,column_c,column_d,column_e,column_f
a1,b1,c1,d1,e1,f1
a2,b2,c2,d2,e2,f2
a3,b3,c3,d3,e3,f3
a4,b4,c4,d4,e4,f4
a5,b5,c5,d5,e5,f5
a6,b6,c6,d6,e6,f6
a7,b7,c7,d7,e7,f7
a8,b8,c8,d8,e8,f8
a9,b9,c9,d9,e9,f9
a10,b10,c10,d10,e10,f10
'

How would I go about reading the file so that I can process and loop through the data?

Andrew
  • 75
  • 1
  • 11
  • `fs.createReadStream()` operate with locale files, so `fileValue` should be a path to the file in the file system. If I understand correctly, `fileValue` is an object with some metadata and a buffer within `buffer` field. – vsemozhebuty Feb 06 '19 at 13:20
  • Ya, my logic may be flawed in treating this as a file, as possibly swagger has now changed it to the object, what would be the best method to now read this object? I have been googling but with no luck as of yet. – Andrew Feb 06 '19 at 13:33
  • Maybe some answers [here](https://stackoverflow.com/questions/13230487/converting-a-buffer-into-a-readablestream-in-nodejs) can help? Or you can try to find a csv parser that uses buffers, not streams. – vsemozhebuty Feb 06 '19 at 13:36

1 Answers1

0

So I was thinking about this the wrong way. As mentioned in the comment fs.createReadStream() is specific to reading from a file. As swagger converts the file to an object, a different method needs to be used.

This was achieved via the csv node module.

npm install csv

And using it like this

const csv = require('csv')

module.exports.uploadPOST = function uploadPOST (req, res, next) { 

  csv.parse(req.swagger.params['upfile'].value.buffer, function (err, data) {
    console.log(data);
  });


};

This will in turn convert the csv into a array of arrays

[ [ 'column_a', 'column_b', 'column_c', 'column_d', 'column_e', 'column_f' ],
  [ 'a1', 'b1', 'c1', 'd1', 'e1', 'f1' ],
  [ 'a2', 'b2', 'c2', 'd2', 'e2', 'f2' ],
  [ 'a3', 'b3', 'c3', 'd3', 'e3', 'f3' ],
  [ 'a4', 'b4', 'c4', 'd4', 'e4', 'f4' ],
  [ 'a5', 'b5', 'c5', 'd5', 'e5', 'f5' ],
  [ 'a6', 'b6', 'c6', 'd6', 'e6', 'f6' ],
  [ 'a7', 'b7', 'c7', 'd7', 'e7', 'f7' ],
  [ 'a8', 'b8', 'c8', 'd8', 'e8', 'f8' ],
  [ 'a9', 'b9', 'c9', 'd9', 'e9', 'f9' ],
  [ 'a10', 'b10', 'c10', 'd10', 'e10', 'f10' ] ]
Andrew
  • 75
  • 1
  • 11