3

I have several hundred GB of backed up folders/files that are stored in an AWS S3 bucket using the "Standard" storage class. I used Arq to back everything up to S3 on a regular basis, automatically.

For a variety of reasons, I chose Standard when I first initiated the backup. However, none of these folders/files need any frequent access any longer, and won't in the future. The Glacier storage class is perfectly suitable now.

I'm aware I can use Lifecycle rules to have AWS automatically scan the bucket every day for files matching certain rules and move them to Glacier. However, I don't really have any need for that -- I'd rather just take the ENTIRE bucket right now, all of it, and move it to Glacier. Just be done with it so that all existing and future files in that bucket are stored in Glacier.

How can I do this? I would greatly prefer to do this through the console or CLI rather than developing a script to use the API. That said, if there's no other way and you are aware of an existing script somewhere, I'm not opposed to it.

Within the S3 console for the bucket, the "Change Storage Class" menu item only provides options for Standard, Standard-IA, One Zone-IA, and Reduced Redundancy.

If you're going to suggest something that requires the CLI, I'd appreciate explicit commands and syntax, if you don't mind -- I'm not familiar enough with the CLI.

Dan
  • 67
  • 2
  • 7

2 Answers2

2

There is a Glacier API you could integrate with: https://docs.aws.amazon.com/amazonglacier/latest/dev/amazon-glacier-api.html but that's a lot of work just to archive one bucket! As @michael-sqlbot mentions below, what you are attempting to achieve is not what this API is really designed for so not a good choice

If you look at previous questions on here about S3/Glacier and moving files between them, most users recommend using lifecycle rules. For example: Move files between amazon S3 to Glacier and vice versa programatically using API

You can setup a lifecycle rule so that it transitions the files from S3 to Glacier 1 day after its creation - so in your case, after you create the rule and it evaluates, it will move all files to Glacier. For example, see this guide (and in particular point 5b): https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html

Jorg Roper
  • 617
  • 1
  • 5
  • 12
  • 1
    Agreed. Using a lifecycle rule is the simplest. Anything else would require iterating through every object to 'copy' the object to itself with a different storage class. – John Rotenstein Jul 29 '18 at 09:15
  • 1
    @JohnRotenstein the Glacier class can't be used with that strategy, anyway. You can only transition to Glacier with lifecycle policies. – Michael - sqlbot Jul 29 '18 at 11:14
  • 3
    @JorgRoper the reference to the Glacier API should probably be removed, or de-emphasized, because that is not comparable to S3 Glacier-class objects. Glacier, used via its API, is an extremely low-level service, designed for building archiving applications -- not something usable directly by humans. You can't, for example, browse the objects once stored via the Glacier API. The S3 Glacier storage class is essentially a case of S3 being an "application" that uses Glacier as a place to archive objects in cold storage. – Michael - sqlbot Jul 29 '18 at 11:22
  • Thank you all. Michael especially, that was really helpful. Both buckets have now been moved. I wasn't entirely clear at first that Lifecycle rules would apply retroactively, but after working it late last night, it finally kicked in. AWS took most of the night to move all the files, but they are all registering in the Console now as having a Glacier storage class. Appreciate the help, folks! – Dan Jul 29 '18 at 17:48
1

This answer likely isn't an appropriate answer for everyone, but depending on your needs, it certainly is an option.

It sounds absurd, but possibly the easiest thing is to just download them out of S3 and then reupload them. This of course only works if there are a handful of them or else you need to write a script.

aws s3 cp s3://[s3-bucket]/[filenames.ext]

aws s3 rm s3://[s3-bucket]/[filenames.ext]

aws s3 cp [filenames.ext] s3://[s3-bucket] --storage-class DEEP_ARCHIVE

Repeat on additional files as necessary.

Of course keep in mind, this could come with transfer in and out fees. For me, this wasn't a big deal.

Andy Borgmann
  • 73
  • 10