2

I would like to upload a form from a web page and directly save the file to S3 without first saving it to disk. This node.js app will be deployed to Heroku, where there is no local disk to save the file to.

The node-formidable library provides a great way to upload files and save them to disk. I am not sure how to turn off formidable (or connect-form) from saving file first. The Knox library on the other hand provides a way to read a file from the disk and save it on Amazon S3.

1) Is there a way to hook into formidable's events (on Data) to send the stream to Knox's events, so that I can directly save the uploaded file in my Amazon S3 bucket? 2) Are there any libraries or code snippets that can allow me to directly take the uploaded file and save it Amazon S3 using node.js?

There is a similar question here but the answers there do not address NOT saving the file to disk.

Community
  • 1
  • 1
rafidude
  • 4,496
  • 7
  • 27
  • 23

2 Answers2

3

It looks like there is no good way to do it. One reason might be that the node-formidable library saves the uploaded file to disk. I could not find any options to do otherwise. The knox library takes the saved file on the disk and using your Amazon S3 credentials uploads it to Amazon.

Since on Heroku I cannot save files locally, I ended up using transloadit service. Though their authentication docs have some learning curve, I found the service useful.

For those who want to use transloadit using node.js, the following code sample may help (transloadit page had only Ruby and PHP examples)

var crypto, signature;
crypto = require('crypto');
signature = crypto.createHmac("sha1", 'auth secret').
    update('some string').
    digest("hex")
console.log(signature);
rafidude
  • 4,496
  • 7
  • 27
  • 23
  • 1
    Speaking as a transloadit cofounder, we added a node example to our docs now too https://transloadit.com/docs/api-docs#authentication – kvz Jan 02 '15 at 15:27
0

this is Andy, creator of AwsSum:

I just released v0.2.0 of this library. It uploads the files that were created by Express' bodyParser() though as you say, this won't work on Heroku:

However, I shall be looking at adding the ability to stream from formidable directly to S3 in the next (v0.3.0) version. For the moment though, take a look and see if it can help. :)

chilts
  • 451
  • 3
  • 12
  • It looks simple enough, though not sure why it wont work on Heroku. Looking forward to v0.3.0 that allows us to directly stream the uploaded file to Amazon S3. This should be event driven and optimized for low memory consumption. – rafidude May 05 '12 at 14:34
  • Agreed, it will be event driven. I'll be looking into the guts of express.bodyParser() which uses formidable, and I'll be copying parts of that. Fingers crossed. – chilts May 07 '12 at 06:53
  • I've decided not to make it stream but instead go from disk, since then it's easier to retry. Even when using it on disk, everything is still streamed so you're not using too much memory: incoming request -> (streamed) to disk -> (streamed) to S3. :) – chilts Nov 01 '12 at 19:54