6

I have a 27GB file that I am trying to move from an AWS Linux EC2 to S3. I've tried both the 'S3put' command and the 'S3cmd put' command. Both work with a test file. Neither work with the large file. No errors are given, the command returns immediately but nothing happens.

s3cmd put bigfile.tsv s3://bucket/bigfile.tsv
waigani
  • 3,570
  • 5
  • 46
  • 71
  • I think the new S3CMD tool supports multipart uploading automatically at least the version it comes in Ubuntu it does it. – Maziyar Nov 20 '13 at 14:30
  • see http://stackoverflow.com/questions/5774808/s3cmd-failed-too-many-times/22107084#22107084, and I suggest: http://stackoverflow.com/a/22107084/116293 – user116293 Mar 01 '14 at 00:29
  • and sometimes it needs root access `sudo s3cmd put...` – tarikakyol Feb 23 '17 at 12:05

3 Answers3

5

Though you can upload objects to S3 with sizes up to 5TB, S3 has a size limit of 5GB for an individual PUT operation.

In order to load files larger than 5GB (or even files larger than 100MB) you are going to want to use the multipart upload feature of S3.

http://docs.amazonwebservices.com/AmazonS3/latest/dev/UploadingObjects.html

http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-upload.html

(Ignore the outdated description of a 5GB object limit in the above blog post. The current limit is 5TB.)

The boto library for Python supports multipart upload, and the latest boto software includes an "s3multiput" command line tool that takes care of the complexities for you and even parallelizes part uploads.

https://github.com/boto/boto

Community
  • 1
  • 1
Eric Hammond
  • 22,089
  • 5
  • 66
  • 75
  • ah, this looks like the next problem I'm about to hit. I'll regenerate the flie and see how I go with s3multiput. Thanks. – waigani Oct 19 '12 at 23:45
0

The file did not exist, doh. I realised this after running the s3 commands in verbose mode by adding the -v tag:

s3cmd put -v bigfile.tsv s3://bucket/bigfile.tsv
waigani
  • 3,570
  • 5
  • 46
  • 71
0

s3cmd version 1.1.0 supports the multi-part upload as part of the "put" command, but its still in beta (currently.)

Suman
  • 9,221
  • 5
  • 49
  • 62