15

In my node.js app, I'm using gm (a graphicsmagick wrapper) and aws2js (amazon cli wrapper). The problem is that Amazon needs to know the content-length so that I can put the stream on S3.

I'm uploading an image to my app, read that file by creating a stream:

var fileStream=fs.createReadStream(file.path)

I pass that file to gm, resize it and then tell it to stream that file. I then want to put that stream to aws:

gm( fileStream, "some.png" ).                               
    identify({bufferStream: true}, function(err, info) {
        this.stream("png", function (err, stdout, stderr) {
            if (err){console.log(err);cb(err);return;}

            aws.S3.putStream(path, stdout, 'public-read', {'content-length': ?????, 'content-type': 'image/png'}, function (err, result) {
            .....       
            }
        });
    });
});

The problem is that Amazon needs to know the content-length (its not the library) to put the stream. AWS doesn't support chunked streams.

Does anyone know how I could determine the content-length of a stream? Or would the only solution be to tmp write it to the disk, read the file as a stream and then put it to amazon with the content-length of the temp file?

JJJ
  • 32,902
  • 20
  • 89
  • 102
japrescott
  • 4,736
  • 3
  • 25
  • 37
  • 1
    Did you ever figure this out? I'm trying to do exactly the same thing... – Bill Aug 07 '12 at 23:48
  • 1
    Bill; I buffer the result and get the size of it. Sadly, there is no other way of doing it, because amazon needs to know the size and a stream is well - a stream - no fixed length. – japrescott Aug 13 '12 at 12:00

2 Answers2

2
gm( fileStream ).filesize({bufferStream: true}, function (error, filesize) {
  this.stream(function (error, stdout, stderr) {
    aws.S3.putStream(path, stdout, 'public-read', {'content-length': filesize, 'content-type': 'image/png'}, function (error, result) {
      // .....       
    });
  });
});
wprl
  • 24,489
  • 11
  • 55
  • 70
1

It depends on how you are doing the upload. I'm using express.js and I'm getting the length from the req.files object like this req.files.[file input name].length.

I'm actually trying to do this exact same thing but I'm having a problem with aws2js recognizing the stream. let me know if you get it working.

jabbermonkey
  • 1,680
  • 4
  • 19
  • 37
  • everything works, if I directly put the stream (since I've got the actual file size, I just uploaded). But since I'm resizing, the resized file is going to have a different content-length, which I need to dermin.. I'll keep you posted.. – japrescott Feb 11 '12 at 16:48
  • Ahh, I forgot about that. I need to do the same thing. And for some reason I'm getting a 400 error from S3. – jabbermonkey Feb 12 '12 at 16:51
  • I guess you are probably using Body Parser with express, and this is why you are having a "req.files" available. Body Parser under the hood will actually save the file first (this is why you can get the size of the file), which I believe was what the OP was trying to avoid doing. You're loosing the advantage of piping the stream directly to S3. – schankam Dec 11 '17 at 13:15