2

I would like to copy a list of files from one S3 bucket to another.

Iterating over the files and using boto3 "bucket.copy" (link) function is very slow.

Is there a way to perform a batch copy or speed things up?

amit
  • 2,171
  • 4
  • 31
  • 50
  • 1
    What do you mean "bucket.copy" is slow? Can you show the copying code? Normally, you should use system wide copy , e.g. s3transfer file copy, instead of reading the content and transfer it. – mootmoot Jul 31 '17 at 08:41
  • @mootmoot, i mean http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.copy – amit Jul 31 '17 at 08:57
  • 2
    Please look into async copy , python asyncio. – mootmoot Jul 31 '17 at 09:01
  • I'll look into it. also found this: https://github.com/aio-libs/aiobotocore – amit Jul 31 '17 at 09:03

1 Answers1

1

The AWS Command-Line Interface (CLI) has a aws s3 sync option that recursively copies new and updated files from the source directory to the destination.

See: Sync documentation

For recursive copy using python, refer to: Boto3 to download all files from a S3 Bucket

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
omuthu
  • 5,948
  • 1
  • 27
  • 37
  • 4
    I would recommend *against* downloading files. You can tell Amazon S3 to perform bucket-to-bucket copies, which is much faster. – John Rotenstein Jul 31 '17 at 11:55