Large AWS uploads failing after random amount of time

Question

With Rails, I've followed this Heroku article to allow direct file uploading to an S3 Bucket. I actually followed this article because my previous implementation wasn't working for multipart uploads (so, large files). Once I implemented this method large files uploaded just fine, except for really large files.

I should note that I strayed from this article a little, in that I'm using v1 of the aws gem, because of our Rails version.

This is how I'm set up:

S3_BUCKET = AWS::S3.new.buckets[ENV['S3_BUCKET_NAME']]

def set_s3_post_url
  @s3_media_post_url = S3_BUCKET.presigned_post(key: "product_media/#{SecureRandom.uuid}-${filename}", success_action_status: '201', acl: 'public-read')
end

As mentioned, this works for large files (~1GB), but when I try to upload one that is, say 10GB, it gets to a mostly uploaded state, then randomly fails. Sometimes after 20 minutes, sometimes after an hour. I thought that maybe the signed URL was expiring, so I explicitly set a long expiry with expires: Time.now + 4.hours, but that didn't seem to work.

I would really appreciate some help with this if anyone has any ideas!

Update

I tried @bbozo's answer of using maxChunkSize, but unfortunately that didn't seem to do it. However, as I was watching the XHR requests in the console, the one that failed returned the following XML response from AWS:

<Error>
    <Code>InternalError</Code>
    <Message>We encountered an internal error. Please try again.</Message>
    <RequestId>1231BD4A29EE5291</RequestId>
    <HostId>f5muQPj2lT2Tmqi49ffqjT4ueLimYvrWUJL6WRW+F7vgm2rL1+FOD3pmsKOEYxFaSFXzLiEZjTg=</HostId>
</Error>

Can you give more info on how exactly it fails? – bbozo Jan 18 '16 at 07:53 — bbozo, Jan 18 '16 at 07:53

bbozo · Answer 1 · 2016-02-14T21:34:01.677

1

Plan A

Did you configure chunked uploads correctly?

https://github.com/blueimp/jQuery-File-Upload/wiki/Chunked-file-uploads

Heroku article doesn't mention anything about setting up chunked uploads in the jquery plugin which handles the file uploads,

the FAQ says:

It is possible to upload files up to 4 GB with the jQuery File Upload plugin. By making use of Chunked file uploads (with chunks smaller than 4GB), the potential file size is unlimited. The restriction of 4 GB is due to some browser limitations, which might be fixed in future updates to those browsers

The documentation for chunked uploads is here, you're supposed to define a maxChunkSize property which defines the size of the chunk

$('#fileupload').fileupload({
    maxChunkSize: 10000000 // 10 MB
});

Plan B

You could put up a minimalistic node.js instance that accepts the file upload and forwards it to S3 via the AWS SDK.

You could also probably run it on a free heroku instance because it probably can be allowed to sleep 6h a day and I've got good experience with file upload services being hosted this way

edited Feb 14 '16 at 21:34

answered Jan 18 '16 at 08:06

bbozo

7,075
3
30
56

Hey @bbozo! Thanks for this suggestion. I tried it out and have updated my question above. – Jody Heavener Jan 19 '16 at 20:28
Np :) Did all of the XHR requests fail or just one of them? – bbozo Jan 20 '16 at 07:29
Whoops missed this. Just the last one (response XML posted above). Still haven't been able to figure it out :( Although I can't accept the answer, you get my expiring bounty points. Thanks! – Jody Heavener Jan 25 '16 at 19:38
@JodyHeavener did you have luck with this in the end? :) – bbozo Feb 13 '16 at 07:53
Nope. Still actively looking for an answer. Tagged this Q with `blueimp` to hopefully grab their attention. If you have anything else I'm all ears! :) – Jody Heavener Feb 14 '16 at 20:33
I fixed your tags a bit, good luck :) will expand the answer too – bbozo Feb 14 '16 at 21:31

score 0 · Answer 2 · answered Jan 18 '16 at 11:11

If you are using the heroku + s3 file upload I will suggest to use the gem 's3_direct_upload'. Using this you can upload the file to AWS::S3 without connecting to heroku server. It creates an direct communication between your server and s3-bucket.

For more details you can use this link

With this you have to use little JavaScript if want to validate the content or make some validations on sizeof file.

score -1 · Answer 3 · answered Jan 15 '16 at 21:53

I work at Signiant as a product manager where we deal exclusively with really large files. I've seen this from some of our other AWS users as well. I think it's an artifact of using TCP / HTTP for uploading really large files.

You may want to consider something other than HTTP for uploading. The product that I manage, Signiant Flight, is UDP based and handles very large files easily. Pricing starts at $12 K / year.

There are other free and open source UDP tools available as well, like UDT (http://udt.sourceforge.net) and Tsunami (http://tsunami-udp.sourceforge.net). To use UDP, you'll need a server in EC2 to connect to, and then have that server write the data to S3.

Flight is a managed file transfer service - we run the servers in the cloud for you, and can achieve speeds as high as 800 - 900 Mbps.

Large AWS uploads failing after random amount of time

3 Answers3

Plan A

Plan B