92

Pretty basic question but I haven't been able to find an answer. Using Transit I can "move" files from one S3 bucket on one AWS account to another S3 bucket on another AWS account, but what it actually does is download the files from the first then upload them to the second.

Is there a way to move files directly from one S3 account to another without downloading them in between?

gak
  • 32,061
  • 28
  • 119
  • 154
Andrew
  • 42,517
  • 51
  • 181
  • 281

12 Answers12

125

Yes, there is a way. And its pretty simple, though it's hard to find it. 8)

For example, suppose your first account username is acc1@gmail.com and second is acc2@gmail.com.

Open AWS Management Console as acc1. Get to the Amazon S3 bucket properties, and in the "Permissions" tab click "Add more permissions". Then add List and View Permissions for "Authenticated Users".

Next, in AWS IAM (it's accessible from among the console tabs) of acc2 create a user with full access to the S3 bucket (to be more secure, you can set up exact permissions, but I prefer to create a temporary user for the transfer and then delete it).

Then you can use s3cmd (using the credentials of the newly created user in acc2) to do something like:

s3cmd cp s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive

All transfer will be done on Amazon's side.

mic
  • 23
  • 6
side2k
  • 2,054
  • 3
  • 17
  • 13
  • 9
    Great tip! Also for syncing whole buckets à la `rsync`, you can use `s3cmd sync s3://acc1_bucket s3://acc2_bucket`. – lllllll Jun 05 '14 at 09:27
  • 6
    If you have problems with a 403 AccessDenied when copying files from target to host, see this post. There may be another way to do it, but the results here worked for me. http://www.alfielapeter.com/posts/8-transferring-s3-bucket-contents-between-accounts-with-s3cmd – crlane Jul 02 '14 at 18:10
  • what about for somewhat large buckets? say in the 50gb area, does it give you any problems? – ianstarz Nov 06 '14 at 16:27
  • 4
    `s3cmd` can be installed with homebrew also, fyi – ianstarz Nov 06 '14 at 16:43
  • 3
    if your destination bucket is meant to serve files publicly.. you may want to add `--acl-public` otherwise, all files are private (at least it was for me when i performed the command). with so many private files, i had to add a policy in the bucket to make them all public again – choonkeat Jan 31 '15 at 01:31
  • Will the transfer add any cost to you? – Henley Jan 06 '16 at 15:13
  • s3cmd has an interactive configuration tool. `s3cmd --configure s3://acc1_bucket` to add credentials for your "acc1_bucket" bucket. – txyoji Sep 13 '16 at 15:35
60

Use the aws cli (I used ubuntu 14 ec2 instance) and just run the following command:

aws s3 sync s3://bucket1 s3://bucket2

You will need to specify the account details for one, and have public write access or public read access to the other.

This will sync the two buckets. You can use the same command again later to sync quickly. Best part is that it doesn't seem t require any bandwidth (e.g. files are not passing through local computer).

Programster
  • 12,242
  • 9
  • 49
  • 55
  • 3
    On a very large bucket I had better luck with this. s3cmd appeared to stall generating the list of files in the first bucket. It may have completed, but I got impatient. – overthink Oct 03 '14 at 18:50
  • 2
    I had the same experience; for whatever reason, `aws s3 sync` seems to run about an order of magnitude faster than `s3cmd cp`. – pix0r Dec 12 '14 at 17:30
  • 3
    This worked for me. If you run into permissions issues, try using the `--grants` option. More info in the documentation: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html – findzen Mar 09 '15 at 21:38
  • 3
    Permissions aren't copied when you sync, so you'll have to specify `--grants`. I learned this after copying a ton of data and then had to redo the sync to a new bucket. `--grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers` sets read permission for public users. – Dan Sandland May 13 '15 at 02:07
  • This ran about 10x faster than `s3cmd` for me, and is also able to sync change files rather than re-copying everything if you need to re-run the command in future - much better! – nfm Jul 06 '15 at 00:10
  • There is a feature request for sync to copy permissions: https://github.com/aws/aws-cli/issues/901 – Programster Aug 17 '15 at 07:41
  • Make sure to copy over the bucket ACLs too - they are not copied over if not specified – iomv Apr 30 '21 at 17:54
22

If you are just looking for a ready made solution there are a few solutions out there that can do this. Bucket Explorer works on Mac and Windows and can copy across accounts as can Cloudberry S3 Explorer and S3 Browser but they are Windows only so may not work for you.

I suspect the AWS console could also do it with the appropriate permissions setup but I haven't tested this.

You can also do it using the AWS API as long as you have given the AWS account you are using write permissions to the destination bucket.

Geoff Appleford
  • 18,538
  • 4
  • 62
  • 85
  • +1 for Bucket Explorer. Much easier than AWS management console or s3cmd for copying a bucket between accounts. The second set of directions worked for me with their free trial: http://www.bucketexplorer.com/documentation/amazon-s3--how-to-copy-files-and-bucket.html – LennonR Jun 22 '13 at 20:03
  • Cloudberry is the way to go. Free, plus it's an easy drag and drop after an even easier setup. And no, I don't work there :) – skidadon May 26 '15 at 14:08
  • 3
    Does Bucket Explorer download and re-upload the data, or does it move the data directly between the Amazon servers? – Wouter Jan 25 '17 at 08:59
  • Note that the trial version of Cloudberry (version 5.0.0.32 at least) has a 5GB transfer limit! – DarthPablo Jul 17 '17 at 08:21
6

boto works well. See this thread. Using boto, you copy objects straight from one bucket to another, rather than downloading them to the local machine and uploading them to another bucket.

Community
  • 1
  • 1
Rose Perrone
  • 61,572
  • 58
  • 208
  • 243
  • 1
    the linked discussion doesn't appear to address the question, as OP is specifically asking about copying from one AWS account to another. – TolkienWASP Oct 21 '20 at 17:14
4

Move S3 files from One account to another account

Let's consider there are two accounts source account and destination account. And two buckets source-bucket and destination bucket. We want to move all files from source-bucket to destination-bucket. We can do it by the following steps:

  1. aws configure
    • Configure your destination account using the credential or the IAM role.
  2. Create user policy for the destination account user.
  3. Give destination user access to the source-bucket by modifying the source-bucket policy and adding destination account user policy into it. By this way, destination user will have the access to source-bucket.
  4. aws s3 ls s3://source-bucket/
    • this will check whether the destination account is having access to source-bucket. Just for confirmation do this.
  5. aws s3 cp s3://source-bucket s3://destination-bucket --recursive
    • this will copy source-bucket all files to destination-bucket. All files are copied using --recursive flag.
  6. aws s3 mv s3://source-bucket s3://destination-bucket --recursive
    • this will move all the files from source-bucket to destination-bucket.

Alternative you can use the sync command - aws s3 sync s3://source-bucket s3://detination-bucket

For Better Explanation follow the link

Anand Tripathi
  • 14,556
  • 1
  • 47
  • 52
3

On Mac OS X I used the Transmit app from Panic. I opened one window for each S3 account (using the API Keys and secrets). I could then drag from one bucket in one window to another bucket in the other window. No need to download files locally first.

Andrew is correct, Transmit downloads the files locally then uploads the files.

One Giant Leap AB
  • 341
  • 3
  • 4
  • 2
    I have Transmit, and I'm 99% positive that the transfer you're describing does download the files to a temp folder and then upload to the other server. Are you positive the files did not flow through your computer? – Andrew Dec 05 '12 at 14:05
  • 2
    I confirm. Transmit downloads and uploads. This makes things awfully slow but has the huge advantage of storing copied contents under the right owner. – Fabio Russo Oct 11 '13 at 09:12
2

CrossFTP can copy S3 files straight from one bucket to another without downloading them. It is a GUI S3 client that works on Windows, Mac, and Linux.

Gatorhall
  • 411
  • 4
  • 7
2

You can user Cyberduck (open source)

Marco Delavegua
  • 61
  • 4
  • In the Windows store they charge something to download this, but from their website you can get it for free. After creating an AWS Access Key ID and Secret Access Key, I have managed to connect to my S3 storage. Didn't test copying anything yet, but looks like it might work. – Wouter Jan 25 '17 at 10:05
1

For newly created files (NOT existing objects), you can take advantage of new functionality from AWS. It is Cross-Region Replication (under "Versioning" for the S3 bucket). You can create a policy that will allow you to replicate new objects to a bucket in a different account.

For existing objects, you will still need to copy your objects using another method - unless AWS introduces native functionality for this in the future.

curious_george
  • 622
  • 2
  • 8
  • 19
0

One can so it with running following :

aws s3 mv (sync for keeping buckets in sync) s3://source-bucket s3://destination-bucket --recursive

  1. Attach a bucket policy to the source bucket in Source Account.

  2. Attach an AWS Identity and Access Management (IAM) policy to a user or role in Destination Account.

  3. Use the IAM user or role in Destination Account to perform the cross-account move.

svikramjeet
  • 1,779
  • 13
  • 27
0

Yes, you can transfer the whole s3 bucket from your root account to another AWS root account.

I have tried the given options but they didn't work for me, even I explore solutions from blogs, but that also didn't work for me. So I started exploring properties and the permission tab in the s3 bucket.

And at last, I find one solution which is very easy to achieve and we do not need to create any IAM role or any policy. Just follow the given steps.

Prerequisites:

  • AWS cli Installed and configured
  • S3 bucket created on both source and destination account

Steps:

  • Navigate to the destination s3 bucket and click on permission tab
  • Scroll down to Access Control List (ACL) and click on Edit button
  • Scroll down to Access for other AWS accounts and click on Add grantee button
  • Enter your canonical ID in the textbox and check the object read and write permission box
  • You can get the canonical ID by clicking on the top right corner of the window at your aws account name -> click on security credentials -> On that page, you can copy your canonical id.
  • After adding the grantee click on the Save Changes button
  • Now open your terminal/cmd and fire below command

aws s3 cp --recursive s3://source-bucket s3://destination-bucket --source-region source-region --region destination-region --acl bucket-owner-full-control

This command will do copy and paste operation but if you want to move then you can use mv instead of cp in above command

Here you can replace source-bucket with your actual bucket name from where you want to copy and replace destination-bucket with your actual bucket name where you want to copy.

You can also specify source and destination region name

You can use your machine to do this or you can spin up one ec2 instance and transfer your s3 data.

Mohmadhaidar devjiyani
  • 459
  • 6
  • 6
0

The given answers so far all require an account that has access to both the source and target s3 buckets. I've found myself recently in a situation where this was not allowed (for various non-technical company reasons that we'll just assume were good).

The solution I ended up going with was to:

  1. Spin up an EC2 instance with permission to write to the target bucket (you can do this locally but the bandwidth and network i/o out of AWS makes EC2 worth it - any tiny instance will do)
  2. Mount the destination folder with s3fs somewhere (/mnt/target)
  3. Give my command line read access (via AWS_ACCESS_KEY_ID, etc) to the source bucket
  4. Use aws sync s3://source_bucket/folder /mnt/target/folder ... (Or mv or cp as needed)

This is the easiest way I've seen to copy between folders when it's not allowed to have a single IAM role with permission to both, and when it's prohibitive to use an intermediate location.

Chipmonkey
  • 863
  • 7
  • 18