2

I am creating a zip file on memory using node-archiver ( https://github.com/archiverjs/node-archiver ). Using archiver's pipe feature, I pipe everything to a Transform stream. I can pipe this stream to a file if needed, but I would like to let Request ( https://github.com/request ) read this stream, so I don't have to get to the filesystem.

Below, my transform stream is called bridge. I am not doing anything special on the Transform (I believe it could also be a PassThrough stream).

var archive = archiver('zip');
archive.pipe(bridge);
var r = request.post(url, function(err, res, body){ .... }
var form = r.form();
form.append('token', <some token>);
form.append('file', bridge, { "filename" : "package.zip", "contentType" : "application/zip" });
archive.append(<some string>, { "name": <some file name> });
archive.finalize();

This doesn't work (it seems the file part is empty). However, if I pipe my bridge Transform stream to a file write stream -- and after stream finishes -- I create a read stream to that file on the form file operation, it works. Of course, because now I have a fully formed file and request can read it (and I wouldn't need a bridge here, just the regular fs stream)

bridge.pipe(fs.createWriteStream(<myfile>));
var archive = archiver('zip');
archive.pipe(bridge);
bridge.on("finish", function(){
    var r = request.post(url, function(err, res, body){ .... }
    var form = r.form();
    form.append('token', <some token>);
    form.append('file', fs.createReadStream(<myfile>), { "filename" : "package.zip", "contentType" : "application/zip" });

}
archive.append(<some string>, { "name": <some file name> });    
archive.finalize();

I wonder if request requires the read stream to be a file stream -- and what else I could be doing wrong here.

noderman
  • 1,934
  • 1
  • 20
  • 36

1 Answers1

1

I have a similar issue and discovered that the problem is neither with nodejs or requestjs.

This is a limitation with HTTP. You cannot upload a file without a known content length, which you cannot know from a transformation until after it's done. See this: HTTP POST: content-length header required?

The only way you can work around this by uploading small chunks at a time, with the readable event. Though, this would require your target api to allow chunked or multipart uploads.

Community
  • 1
  • 1
thejinx0r
  • 444
  • 7
  • 10
  • That was my fear. So, although Request accepts a stream (fs.createReadStream), it is getting it's content-length from the readstream object. I wonder if it is possible to get the archiver content-length upon finalize and then set this header for Request. It seems it could be possible by listening for a close event on the archiver (similar to this http://stackoverflow.com/a/30882016/2020565 -- and/or using the archiver pointer https://archiverjs.com/docs/Archiver.html#pointer to accrue the total bytes. – noderman Jul 22 '16 at 22:31
  • 1
    You would still need to know the content-length prior to initiating the HTTP request. So I don't see how that would help. One of things you could do is create a passthrough stream, with a large enough highWaterMark, to store your zip in memory and then uploading that new stream. That was not a possibility in my case as I was expecting streams that could be on the order of 10GBs. – thejinx0r Jul 23 '16 at 11:30
  • 1
    Maybe it would be possible to "pipe the archiver to null", get the content-size and **then** redo the operation, this time setting the content-length with the length data acquired. **Obviously**, this would mean performing an archive operation twice, but would avoid the need to store everything in memory. It would be good to have this alternative. I'll try that just for kicks. – noderman Jul 23 '16 at 17:06