12

I have read-only access to a source S3 bucket. I cannot change permissions or anything of the sort on this source account and bucket. I do not own this account.

I would like to sync all files from the source bucket to my destination bucket. I own the account that contains the destination bucket.

I have a separate sets of credentials for the source bucket that I do not own and the destination bucket that I do own.

Is there a way to use the AWS CLI to sync between buckets using two sets of credentials?

aws s3 sync s3://source-bucket/ --profile source-profile s3://destination-bucket --profile default

If not, how can I setup permissions on my owned destination bucket to that I can sync with the CLI?

duffn
  • 3,690
  • 8
  • 33
  • 68
  • Are the two buckets in different AWS accounts? Do you have the ability to modify bucket policies? If so see this answer: http://stackoverflow.com/a/17162973/1428388 – jzonthemtn Aug 08 '16 at 13:50
  • @jbird They are not. They are in different accounts. The source bucket in an account I do not own, but have been given credentials to read, and the destination bucket in my owned account. – duffn Aug 08 '16 at 13:51
  • 4
    It might be quickest to use an EC2 instance to download the files from one bucket and upload them to the other bucket. That way you don't have to pay for the bandwidth charges and get better speeds. There could exist a better method but I'm not aware of it. – jzonthemtn Aug 08 '16 at 13:53

3 Answers3

6

The built-in S3 copy mechanism, at the API level, requires the request be submitted to the target bucket, identifying the source bucket and object inside the request, and using a single set of credentials that has both authorization to read from the source and write to the target.

This is the only supported way to copy from one bucket to another without downloading and uploading the files.

The standard solution is found at http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html.

You can grant their user access to write your bucket or they can grant your user access to their bucket... but copying from one bucket to another without downloading and re-uploading the files is impossible without the complicity of both account owners to establish a single set of credentials with both privileges.

Michael - sqlbot
  • 169,571
  • 25
  • 353
  • 427
3

Use rclone for this. It's convenient but it does download and upload the files I believe which makes it slow for large data volumes.

rclone --config=creds.cfg copy source:bucket-name1/path/ target:bucket-name2/path/

creds.cfg:

[source]
type = s3
provider = AWS
access_key_id = AAA
secret_access_key = bbb

[target]
type = s3
provider = AWS
access_key_id = CCC
secret_access_key = ddd
themadmax
  • 2,344
  • 1
  • 31
  • 36
citynorman
  • 4,918
  • 3
  • 38
  • 39
  • 1
    This is easily the most convenient option and possibly the only one if you don't own one of the two accounts. It contains a type though: `destination` in the command should be `target` according to the config file. – Andreas Dec 28 '22 at 02:14
1

For this use case, I would consider Cross-Region Replication Where Source and Destination Buckets Are Owned by Different AWS Accounts

... you set up cross-region replication on the source bucket owned by one account to replicate objects in a destination bucket owned by another account.

The process is the same as setting up cross-region replication when both buckets are owned by the same account, except that you do one extra step—the destination bucket owner must create a bucket policy granting the source bucket owner permission for replication actions.

Rodrigo Murillo
  • 13,080
  • 2
  • 29
  • 50