1

My issue is to get image uploading to amazon working. I was looking for a solution that doesnt save the file on the server and then upload it to Amazon.

Googling I found pkgcloud and on the README.md it says:

Special attention has been paid so that methods are streams and pipe-capable.

Can someone explain what that means and if it is what I am looking for?

Kyle Kelley
  • 13,804
  • 8
  • 49
  • 78
piggyback
  • 9,034
  • 13
  • 51
  • 80

1 Answers1

3

Yupp, that means you've found the right kind of s3 library.

What it means is that this library exposes "streams". Here is the API that defines a stream: http://nodejs.org/api/stream.html

Using node's stream interface, you can pipe any readable stream (in this case the POST's body) to any writable stream (in this case the S3 upload).

Here is an example of how to pipe a file upload directly to another kind of library that supports streams: How to handle POSTed files in Express.js without doing a disk write

EDIT: Here is an example

var pkgcloud = require('pkgcloud'),
      fs = require('fs');

var s3client = pkgcloud.storage.createClient({ /* ... */ });

app.post('/upload', function(req, res) {
  var s3upload = s3client.upload({
    container: 'a-container',
    remote: 'remote-file-name.txt'
  })

  // pipe the image data directly to S3
  req.pipe(s3upload);      
});

EDIT: To finish answering the questions that came up in the chat:

req.end() will automatically call s3upload.end() thanks to stream magic. If the OP wants to do anything else on req's end, he can do so easily: req.on('end', res.send("done!"))

Community
  • 1
  • 1
rdrey
  • 9,379
  • 4
  • 40
  • 52
  • will it be saved in memory then? – piggyback Feb 06 '13 at 10:07
  • It won't be "saved" but it will stay in memory during the `pipe()`, yes. :) – rdrey Feb 06 '13 at 13:27
  • argh. you think that is there any "better way" to do file uploading to amazon? – piggyback Feb 06 '13 at 15:26
  • hehe. that's a completely different question ;) but here is an alternative, that you might think is "better": http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html – rdrey Feb 06 '13 at 15:33
  • 1
    but to clarify: using the code above, your users will be able to upload large files, since only parts of the files will ever be in memory at a time. does that help? – rdrey Feb 06 '13 at 15:41
  • hold on, I dont understand on thing. How do I close the post request if there is no req.end ? and where do I take the file the user uploaded? – piggyback Feb 06 '13 at 15:49
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/24044/discussion-between-rdrey-and-haskellguy) – rdrey Feb 06 '13 at 15:54