Piping stream from busboy to request post

Question

I have multipart/form-data that I am posting to an express endpoint /data/upload, form markup below:

form(enctype="multipart/form-data", action="/data/upload", method="post")
  input(type="file", name="data")

I'm using busboy to read the file stream, which is working fine. From there, I want to send the stream again as multipart/form-data to a second Java backend, using the request npm module. JS client/Java server code below:

  req.busboy.on('file', function (fieldName, fileStream, fileName, encoding, mimeType) {

    var reqBody = {
      url: server.baseURL + 'api/data',
      headers: {
        'Connection': 'keep-alive',
        'Content-Type': 'multipart/form-data'
      },
      formData: {
        file: fileStream
      }
    };

    request.post(reqBody, function (err, r, body) {
      // Do rendering stuff, handle callback
    });
 });

Java endpoint (api/data)

@POST
@Consumes(MediaType.MULTIPART_FORM_DATA)
public void addData(FormDataMultiPart formDataMultiPart) {
  // Handle multipart data here      
}

I don't think I'm sending the file correctly as multipart/form-data here... but I'm having a hard time figuring out how to essentially pipe the stream from busboy directly to request without reading/writing from a temp file on the client-side. Any ideas?

Java stack trace:

Apr 27, 2016 5:07:12 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 3 * Server has received a request on thread qtp1631904921-24
3 > POST http://localhost:8080/api/data
3 > Connection: keep-alive
3 > Content-Length: 199
3 > Content-Type: multipart/form-data; boundary=--------------------------331473417509479560313628
3 > Host: localhost:8080

Apr 27, 2016 5:07:12 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 3 * Server responded with a response on thread qtp1631904921-24
3 < 400

17:07:13.003 [qtp1631904921-24] WARN  org.eclipse.jetty.http.HttpParser parseNext - bad HTTP parsed: 400 No URI for HttpChannelOverHttp@425137da{r=1,c=false,a=IDLE,uri=null}

Rahat's recommended change:

 31     var reqBody = {
 32       url: server.baseURL + 'data',
 33       headers: {
 34         'Connection': 'keep-alive',
 35         'Content-Type': 'multipart/form-data'
 36       }
 37     };
 38 
 39     req.pipe(req.busboy.pipe(request.post(reqBody)));

Threw error:

Error: Cannot pipe. Not readable.
   at Busboy.Writable.pipe (_stream_writable.js:154:22)

It's a poor choice of variable name, but it's not a problem. I'll change it in the post above. — sir_thursday, Apr 27 '16 at 20:56
Have you looked into piping your `busboy` to your java endpoint? — Rahat Mahbub, Apr 27 '16 at 20:58
Do you have a code snippet or link to documentation? Thanks-- — sir_thursday, Apr 27 '16 at 21:00
Rahat I tried your code, seems like there's an issue with chaining the streaming... see http://stackoverflow.com/questions/17098400/how-to-chain-write-stream-immediately-with-a-read-stream-in-node-js-0-10 — sir_thursday, Apr 27 '16 at 21:14
File is a ReadableStream. So, how about this: `fileStream.pipe(request.post(url))` — Rahat Mahbub, Apr 27 '16 at 21:18
Hm. Slightly different (albeit similar) error thrown. `5 > POST http://localhost:8080/api/data 5 > Connection: keep-alive 5 > Content-Type: multipart/form-data 5 > Host: localhost:8080 5 > Transfer-Encoding: chunked Apr 27, 2016 5:22:42 PM org.glassfish.jersey.filter.LoggingFilter log INFO: 5 * Server responded with a response on thread qtp1991537156-17 5 < 400` — sir_thursday, Apr 27 '16 at 21:23
There is no enough information in the log. Can you set log level to debug or even more detailed? — Ilya, Apr 27 '16 at 21:48

Afanasii Kurakin · Accepted Answer · 2016-04-28T01:31:35.090

The problem here is that you need to provide 'Content-Length' for the multipart upload manually, because request (and underlying form-data) can't figure it out by themselves. So request sends invalid Content-Length: 199 (the same for any incoming file size), which breaks the java multipart parser.

There are multiple workarounds:

1) Use incoming request 'Content-Length'

request.post({
  url: server.baseURL + 'api/data',
  formData: {
    file: {
      value: fileStream,
      options: {
        knownLength: req.headers['content-length']
      }
    }
  }
}, function (err, r, body) {
  // Do rendering stuff, handle callback
})

This will produce a bit incorrect request though, because incoming length includes other upload fields and boundaries, but busboy was able to parse it w/o any complaints

2) Wait until file is completely buffered by the node app then send it to java

var concat = require('concat-stream')
req.busboy.on('file', function (fieldName, fileStream, fileName, encoding, mimeType) {
  fileStream.pipe(concat(function (fileBuffer) {
    request.post({
      url: server.baseURL + 'api/data',
      formData: {
        file: fileBuffer
      }
    }, function (err, r, body) {
      // Do rendering stuff, handle callback
    })
  }))
})

This will increase app memory consumption, so you needed to be careful and consider using busboy limits

3) Buffer file to disk before uploading (just for the reference)

express + multer - I recommend using express for webservers, it makes things more manageable, and multer is based on the busboy
formidable

Just tried it- seems to work. Thank you so much for taking the time to write this up. I'll award bounty as soon as it lets me. — sir_thursday, Apr 28 '16 at 02:12
You are welcome! Another lib that may be useful for your case (node - java integration) - is https://github.com/nodejitsu/node-http-proxy to proxy some requests to java directly, plus I'd suggest to take a look at https://github.com/visionmedia/superagent as a replacement for 'request' - it has more clear syntax and can run on both browser and backend. — Afanasii Kurakin, Apr 28 '16 at 03:01
In my tests 2 still does not works for some small files. Consider send a custom header with exact file size, which could always be read before stream start. — felipeaf, Jan 15 '18 at 16:27

score 0 · Answer 2 · answered Jan 15 '18 at 16:32

If possible, send a custom header with exactly size file (bytes). The header always can be read before handle payload stream. Use this instead of content-length header of previous answer, because that sometimes doesn't works (with small files, i guess, but i cannot ensure that works with large files).

score 0 · Answer 3 · answered May 23 '19 at 03:11

For the answer of Afanasii Kurakin

request.post({
  url: server.baseURL + 'api/data',
  formData: {
    file: {
      value: fileStream,
      options: {
        knownLength: req.headers['content-length']
      }
    }
  }
}, function (err, r, body) {
  // Do rendering stuff, handle callback
})

You should change from req.headers['content-length'] to the real file size, normally the content-length from header is bigger than the file size. I got pain because of the content-length and after using the file size, everything worked perfectly.

Piping stream from busboy to request post

3 Answers3

Linked