Rails - Resave All Models for S3 Migration

Question

rails 6.1.3.2 aws-sdk-s3 gem

I currently have a rails app in production that uses ActiveStorage to attach image data to a wrapper Image model. It's currently using the local strategy to save images to disk and I am migrating it to S3. I am not using paperclip or anything similar.

I succeeded in setting it up. Currently it is set to use local primarily and have S3 as a mirror so that I can write to two places during the migration. However the documentation says that it will only save new images to S3 upon create and update of a record. I would like to "re-save" all models in production to force the migration to happen. Does anyone know how to do this?

devleo · Accepted Answer · 2021-06-25T15:11:24.967

Looks like it was already answered!

If you happen to be stuck with only access to the Rails Console like I was, this solution worked perfectly. If you copy-paste this code into the console, it will begin to produce output of the S3 uploads. After 5k of those, I was done. An immense thank you to Tayden for the solution.

all_services = [ActiveStorage::Blob.service.primary, *ActiveStorage::Blob.service.mirrors]

    # Iterate through each blob
    ActiveStorage::Blob.all.each do |blob|

      # Select services where file exists
      services = all_services.select { |file| file.exist? blob.key }

      # Skip blob if file doesn't exist anywhere
      next unless services.present?

      # Select services where file doesn't exist
      mirrors = all_services - services

      # Open the local file (if one exists)
      local_file = File.open(services.find{ |service| service.is_a? ActiveStorage::Service::DiskService }.path_for blob.key) if services.select{ |service| service.is_a? ActiveStorage::Service::DiskService }.any?

      # Upload local file to mirrors (if one exists)
      mirrors.each do |mirror|
        mirror.upload blob.key, local_file, checksum: blob.checksum
      end if local_file.present?

      # If no local file exists then download a remote file and upload it to the mirrors (thanks @Rystraum)
      services.first.open blob.key, checksum: blob.checksum do |temp_file|
        mirrors.each do |mirror|
          mirror.upload blob.key, temp_file, checksum: blob.checksum
        end
      end unless local_file.present?

score 0 · Answer 2 · answered May 01 '23 at 11:50

Since you are already on rails 6.1, you can use #mirror_later. It will enqueue a background worker to mirror to the configured service.

ActiveStorage::Blob.find_each do |blob|
  blob.mirror_later
end

Reference: https://github.com/rails/rails/blob/912096d4ce930b8e7e5d91e0c86bae2091fda0e4/activestorage/app/models/active_storage/blob.rb#L300-L302

Rails - Resave All Models for S3 Migration

2 Answers2