1

I would like to upload files of users to S3. I have a web client/NodeJS Express solution.

So far in my research i found that using multipart uploads, i can do either one of these:

1.) If streaming is enabled directly using pipes, then i cannot control the size of the file.

2.) If size needs to be controlled, pipe/streaming has to be disabled and file has to be written to disk or memory buffer to check for file size then transfer it to S3.

Is there any way i can get both of these ?

I am looking at the following middle wares :

busboy

multiparty

multer

formidable

Community
  • 1
  • 1
Koder
  • 1,794
  • 3
  • 22
  • 41

1 Answers1

1

At least with busboy, you can set various request-wide limits. So you could set a file size limit for all files (or do it on a per-file basis by doing the counting yourself and draining any extra data over your custom file size limit), then stream directly to S3 (using Amazon's multipart API). If busboy notifies you that the file size limit was reached, then you'd simply have to delete the partial file you just streamed to S3.

Otherwise the alternative is like you said, to save to disk first and then upload to S3.

Both have their advantages and disadvantages, but because of the nature of multipart streams, there really are no other solutions to this particular problem.

mscdex
  • 104,356
  • 15
  • 192
  • 153