1

I have been struggling in the last few days to get this working.

Basically I have a form in HTML that has a type="file" input.

I would like that this file is sent to nodejs and directly streamed to Amazon s3. In this way I will use less storage on my server (imagine if 100 people upload 1g, at the end my server will have 100g), so if I stream I will occupy just memory for a little amount of time

Any hint on how to do this with express?

piggyback
  • 9,034
  • 13
  • 51
  • 80

1 Answers1

1

I am unsure if you're looking for a CORS approach, where the user uploads directly to your S3 bucket, or an approach where you use express as a middle-man.

If you're looking at a CORS approach (which I would recommend unless you are doing anything special with the file before it gets put in S3 that can't be done client side), this should help you out: http://bencoe.tumblr.com/post/30685403088/browser-side-amazon-s3-uploads-using-cors

If you are planning to use your server as a mid-point (ie, upload to your server, then upload from your server to S3), I have heard good things about Knox (https://github.com/LearnBoost/knox). I've played around with it, but haven't actually used it as I tend to use CORS approaches.

Nick Mitchinson
  • 5,452
  • 1
  • 25
  • 31
  • I think that there is a third way which is "stream" the file while you get it node directly to amazon, isnt it? – piggyback Feb 06 '13 at 17:27
  • Any info here (http://stackoverflow.com/questions/12526673/node-application-stream-file-upload-directly-to-amazon-s3) you may find use ful, as your question seems to be a duplicate. As for the third options, I am not quite sure as it's not something I've ever looked into. If you are going to do that you may as well just use CORS and take the load off your server entirely. – Nick Mitchinson Feb 06 '13 at 17:34
  • Well I am actually looking for the third option. :S Hold on, if I use CORS wouldnt it be complicate to check the state of the transaction from the backend? How do I know what is the address of the link amazon generated? How do I know if the user manipulated that? I feel it is sort of decrease of control.. – piggyback Feb 06 '13 at 18:22
  • 1
    What I do is as follows: 1) user attempts to upload. I make a request to my server for S3 credentials. This returns the 'manifest' as its referred to in the article. It also generates a file path for the location on S3 (which is used as the key), and inserts all the file information into the db, flagged as incomplete. 2) When the client gets this, it initiates the CORS upload. 3) When the uplaod is complete, make a request to my server which updates the db row as complete. 4) At this point, alert the user that the uplaod was successful – Nick Mitchinson Feb 06 '13 at 18:33
  • Sorry to keep asking questions, but once you upload, how can you ensure that in point 3 the user doesn't influence the request? For example no internet connection => you have the file but is not on db, or the user change params in the request :/ – piggyback Feb 06 '13 at 18:43
  • 1
    Its a signed request. When you generate the credentials on your server, you have your credential object which includes this such as the key (path it will save as), times its valid from, permissions to upload it with etc. Your response also includes a signature, which is a base64 encoding of the object using your AWS Secret Key. I can't remember right now exactly how to do it but it's explained in the docs. AWS then encodes the object with your key (only you and AWS should know this key). If it doesnt match (because, say, the user changed something), then the request is rejected. – Nick Mitchinson Feb 06 '13 at 18:48