uploading folder to s3 with ruby sdk

Question

I have a script that is supposed to upload a local folder to s3 using aws-sdk and ruby. As much as I understand from ruby, the files need to be uploaded one by one, so here is the code used:

require 'aws-sdk'
require 'open3'
s3_bucket = ARGV[0]
debug = ARGV[1] || nil
@s3 = Aws::S3::Client.new(region: 'eu-west-1')
files = Dir[ File.join('srv', '**', '*') ].reject { |p| File.directory? p }
files.each do |f|
    o, e, s = Open3.capture3("gio info -a standard::content-type #{f}")
    abort(e) unless s.to_s.match(/exit 0/)
    content_type = o.split('standard::content-type: ')[1].strip
    s3_key = f.split('srv/lila/')[1]
    puts "Uploading #{f} with content-type #{content_type}" if debug
    File.open(f,'rb') do |file|     
        @s3.put_object({body: file, content_type: content_type, bucket: s3_bucket, key: s3_key})
    end
end

My local file name is like this: srv/lila/1.1.1/somename/index.html Somehow, only the file name is uploaded, and not the content.So when I go to the URL, I can see the name of the file as the content srv/lila/1.1.1/somename/index.html. My ruby knowledge is limited and I am not sure what is wrong in this script. Can you help please?

engineersmnky · Answer 1 · 2021-04-20T17:17:23.367

2

Your issue is this line:

resp = @s3.put_object({body: f, content_type: content_type, bucket: s3_bucket, key: s3_key})

In this case f is not a File but rather a String that represents the path to a file.

body: accepts a String, StringIO or File object as an argument. In this case you are passing a String and it treats that as the contents of the uploaded file.

Instead I would recommend the following alteration:

  File.open(f,'rb') do |file| 
    @s3.put_object({body: file, content_type: content_type, bucket: s3_bucket, key: s3_key})
  end

Now file is an actual File object.

I also removed resp as that local variable did not serve a purpose.

edited Apr 20 '21 at 17:17

answered Apr 20 '21 at 17:05

engineersmnky

25,495
2
36
52

Thank you, I understand what you are trying to do. I have updated my script. The content of the file is still showing the file name and the file variable in the puts command shows this: `Uploading # with content-type image/png` Does it look correct to you? – Em774 Apr 20 '21 at 17:46
@Em774 I was simply suggesting replacing that one line ( `resp = @s3.put_object({body: f, content_type: content_type, bucket: s3_bucket, key: s3_key})`) with the 3 lines I posted. The rest is fine as is. You could also try `resp = @s3.put_object({body: File.read(f, mode: 'rb'), content_type: content_type, bucket: s3_bucket, key: s3_key})` This will read the entire file contents into memory though which might be an issue – engineersmnky Apr 20 '21 at 17:47
thank you I have tried the first one and pasted the updated code above for better readability. I still don't get the picture I need. Now I currently have a blank page instead. – Em774 Apr 20 '21 at 18:07
sorry for the other post, I have changed the original code instead. I have tried with `rb` too, it still gives me the name of the file instead of the file itself. – Em774 Apr 20 '21 at 18:21
So you are saying that the contents of the uploaded file is simply the name of the file, right? I cannot understand why this would be the case based on the [Documentation](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_object-instance_method) – engineersmnky Apr 20 '21 at 19:58
I just wanted to come back to you to tell you what worked for me in the end: `IO.read(f)` made it work and invalidating the cache later on. The documentation says something like this ```resp = client.put_object({ body: "HappyFace.jpg", bucket: "examplebucket", key: "HappyFace.jpg", })``` so it was quite confusing. Thank you for your precious help. – Em774 Apr 21 '21 at 11:02

uploading folder to s3 with ruby sdk

1 Answers1