2

I have requirement where I need to copy files from one S3 bucket to other S3 bucket. These buckets are present in different AWS account. I tried using s3 sync command. But, for this destination IAM user should be given with read access on source bucket. I am not using this command because I don't have same same IAM user with access to both the accounts. There are two AIM user as of now, one for each.

Is there anyway I can use s3-dist-cp command for this requirement?

Kishor Baindoor
  • 111
  • 3
  • 9

1 Answers1

4

As of this writing, you cannot configure two different accesskeys for two different buckets.

You can configure the same IAM user across accounts and allow them to read from one and write to another. IAM roles can make it happen here.

If you don't want to use the same IAM user across accounts copy to local hdfs and move back to destination bucket.

hadoop distcp -Dfs.s3a.awsAccessKeyId= -Dfs.s3a.awsSecretAccessKey= s3a://sourcebucket hdfs://tempcopy/

hadoop distcp -Dfs.s3a.awsAccessKeyId= -Dfs.s3a.awsSecretAccessKey= hdfs://tempcopy/ s3a://destbucket

Hope it helps.

Kannaiyan
  • 12,554
  • 3
  • 44
  • 83