How to copy file across buckets using aws-s3 or aws-sdk gem in ruby on rails

Question

The aws-s3 documentation says:

  # Copying an object
  S3Object.copy 'headshot.jpg', 'headshot2.jpg', 'photos'

But how do I copy heashot.jpg from the photos bucket to the archive bucket for example

Thanks!

Deb

score 28 · Answer 1 · edited Oct 14 '22 at 17:47

AWS-SDK gem. S3Object#copy_to

Copies data from the current object to another object in S3.
S3 handles the copy so the client does not need to fetch the 
data and upload it again. You can also change the storage 
class and metadata of the object when copying.

It uses copy_object method internal, so the copy functionality allows you to copy objects within or between your S3 buckets, and optionally to replace the metadata associated with the object in the process.

Standard method (download/upload)

enter image description here

Copy method

enter image description here

Code sample:

require 'aws-sdk'

AWS.config(
  :access_key_id     => '***',
  :secret_access_key => '***',
  :max_retries       => 10
)

file     = 'test_file.rb'
bucket_0 = {:name => 'bucket_from', :endpoint => 's3-eu-west-1.amazonaws.com'}
bucket_1 = {:name => 'bucket_to',   :endpoint => 's3.amazonaws.com'}

s3_interface_from = AWS::S3.new(:s3_endpoint => bucket_0[:endpoint])
bucket_from       = s3_interface_from.buckets[bucket_0[:name]]
bucket_from.objects[file].write(open(file))

s3_interface_to   = AWS::S3.new(:s3_endpoint => bucket_1[:endpoint])
bucket_to         = s3_interface_to.buckets[bucket_1[:name]]
bucket_to.objects[file].copy_from(file, {:bucket => bucket_from})

Be awared of AWS S3 limit: 100 request/sec across all regions for upload/download both calls. — Anatoly, Apr 09 '14 at 08:57

score 6 · Accepted Answer · edited May 23 '17 at 12:25

Using the right_aws gem:

# With s3 being an S3 object acquired via S3Interface.new
# Copies key1 from bucket b1 to key1_copy in bucket b2:
s3.copy('b1', 'key1', 'b2', 'key1_copy')

the gotcha I ran into is that if you have pics/1234/yourfile.jpg the bucket is only pics and the key is 1234/yourfile.jpg

I got the answer from here: How do I copy files between buckets using s3 from a rails application?

rigdonmr · Answer 3 · 2017-09-06T22:24:53.803

6

For anyone still looking, AWS has documentation for this. It's actually very simple with the aws-sdk gem:

bucket = Aws::S3::Bucket.new('source-bucket')
object = bucket.object('source-key')

object.copy_to(bucket: 'target-bucket', key: 'target-key')

edited Sep 06 '17 at 22:24

answered Sep 06 '17 at 21:42

rigdonmr

2,662
3
26
40

score 3 · Answer 4 · answered Jul 19 '13 at 13:44

3

When using the AWS SDK gem's copy_from or copy_to there are three things that aren't copied by default: ACL, storage class, or server side encryption. You need to specify them as options.

from_object.copy_to from_object.key, {:bucket => 'new-bucket-name', :acl => :public_read}

https://github.com/aws/aws-sdk-ruby/blob/master/lib/aws/s3/s3_object.rb#L904

answered Jul 19 '13 at 13:44

Greg Benedict

219
2
5

Are you sure this method covers the question? The author specifically asks for a copy between two buckets. – Tim Lawrenz Dec 19 '20 at 12:44

score 2 · Answer 5 · answered May 10 '16 at 23:22

2

Here's a simple ruby class to copy all objects from one bucket to another bucket: https://gist.github.com/edwardsharp/d501af263728eceb361ebba80d7fe324

answered May 10 '16 at 23:22

esharp

1,317
1
12
16

score 2 · Answer 6 · answered Feb 07 '20 at 10:57

Multiple images could easily be copied using aws-sdk gem as follows:

require 'aws-sdk'

image_names = ['one.jpg', 'two.jpg', 'three.jpg', 'four.jpg', 'five.png', 'six.jpg']
Aws.config.update({
    region: "destination_region", 
    credentials: Aws::Credentials.new('AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY')
})
image_names.each do |img|
    s3 = Aws::S3::Client.new()
    resp = s3.copy_object({
      bucket: "destinationation_bucket_name", 
      copy_source: URI.encode_www_form_component("/source_bucket_name/path/to/#{img}"), 
      key: "path/where/to/save/#{img}" 
    })      
end

If you have too many images, it is suggested to put the copying process in a background job.

score 0 · Answer 7 · answered Feb 01 '13 at 00:19

I ran into the same issue that you had, so I cloned the source code for AWS-S3 and made a branch that has a copy_to method that allows for copying between buckets, which I've been bundling into my projects and using when I need that functionality. Hopefully someone else will find this useful as well.

View the branch on GitHub.

score 0 · Answer 8 · answered Aug 11 '10 at 14:40

0

I believe that in order to copy between buckets you must read the file's contents from the source bucket and then write it back to the destination bucket via your application's memory space. There's a snippet showing this using aws-s3 here and another approach using right_aws here

answered Aug 11 '10 at 14:40

bjg

7,457
1
25
21

I'd like to use the snippet for aws-s3 but I'm having problems initializing the AmazoneS3Asset class. Do I simply put the snippet that defines the class in config/initializers ? – deb Aug 11 '10 at 17:38
I found the answer to my comment here http://stackoverflow.com/questions/1146946/ruby-on-rails-and-external-classes – deb Aug 11 '10 at 17:47
apparently right_aws has this fucntionality, I didn't need to add the snippet. I posted more info in the answer below – deb Aug 11 '10 at 18:24

score 0 · Answer 9 · answered Aug 18 '11 at 23:55

0

The aws-s3 gem does not have the ability to copy files in between buckets without moving files to your local machine. If that's acceptable to you, then the following will work:

AWS::S3::S3Object.store 'dest-key', open('http://url/to/source.file'), 'dest-bucket'

answered Aug 18 '11 at 23:55

Jared

2,885
2
28
31

aws supports now this, the gem also. – Ivailo Bardarov Dec 14 '18 at 15:59
Yes, this answer is now outdated. – Jared Dec 21 '18 at 20:36

How to copy file across buckets using aws-s3 or aws-sdk gem in ruby on rails

9 Answers9

Linked