2

I am trying to copy from one bucket to another bucket and each bucket has their own access key and secret.

I can connect to the first bucket and down load a file just fine. It might be important to note that I do not have full access to the bucket I am copying from, meaning I can not read all keys in the bucket, just a subset I have access to. I have complete control on the second bucket I am copying to.

client2 is where I am copying to and client is where I am copying from.

copy_source = {
    'Bucket': bucketName,
    'Key': key
     }

client2.copy(CopySource = copy_source,Bucket=bucketName2,Key=key,SourceClient=client)

Here is the error I get: botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the UploadPartCopy operation: Access Denied

I am a newbie and any help would be greatly appreciated!!

signasl
  • 23
  • 1
  • 3
  • 1
    I *think* you will find that this occurs because `SourceClient` doesn't exactly do what you may be assuming it does. It apparently isn't used to read the source object, but only to discover some source metadata that is required for copying large objects in consistent parts. The S3 API's PUT+copy operation requires that the invoking client's credentials have privileges on both buckets, thus permitting the copy to be accomplished internally to S3, avoiding the time, bandwidth, and cost of download + upload. Not possible unless you have a single set of credentials with access to both buckets. – Michael - sqlbot Mar 25 '18 at 02:09
  • ...but my expertise is with the operation of the actual S3 REST API that boto3 and the other SDKs use to communicate with the S3 service, rather than specifically with boto3, so I could be wrong, but note that [*"`SourceClient` (botocore or boto3 Client) -- The client to be used for operation that may happen at the source object. For example, this client is used for the head_object that determines the size of the copy"*](http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.copy) doesn't mention `SourceClient` being used for reading/downloading. – Michael - sqlbot Mar 25 '18 at 02:14

1 Answers1

5

The reason you're likely getting the Access Denied on this is because the SourceClient is only used for getting the size of the object to determine if it can be copied directly, or if a multi-part upload is required.

When it comes to the actual copy itself, the underlying the underlying copy_object method on the client, which does not accept a SourceClient, and calls out to the S3 APIs PUT Object - Copy method.

As such, if you want to be able to perform an S3 copy from one bucket to another, you can either give the user associated with the access key used by client2 permission to read from the Source bucket, or you can perform an S3 Get using client1 then an S3 Put with client2.

Jamie Starke
  • 8,776
  • 3
  • 38
  • 58
  • Thanks! Unfortunately I don't control the source bucket it's a third-party application that I only have access to read specific files in the bucket. I'll look at the get and put option to see if that works. – signasl Mar 25 '18 at 17:30
  • Thanks for the direction. I can 'get' just fine but I can't just 'put' the object retrieved in the get. The file is a .gz and body of the object returned is "botocore.response.StreamingBody". Any direction on how to actually get the file to move? – signasl Mar 26 '18 at 00:34
  • Totally. Based on http://botocore.readthedocs.io/en/latest/reference/response.html, if you call the `read()` method, with no arguments, it should read the entire body as bytes. – Jamie Starke Mar 26 '18 at 00:37
  • Thanks for patients with a newbie. That is what I tried first. I do a get, and then I did a read (I can see data if I print it), but when I do a put, a file gets created in my other bucket, but there is no data. The file size is zero, even thought the source is fairly large. s3.Object(bucketName2,'streamBody').put(streamBody) – signasl Mar 26 '18 at 00:46
  • Try [upload_fileobj](http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Object.upload_fileobj) without doing the read first. – Jamie Starke Mar 26 '18 at 03:46