17

I am creating a PHP based web application using Amazon's S3 and glacier services.

Now I want to give my site users a feature that they can choose any file and make it archive (means move file from S3 to Glacier) and unarchive (means move file from Glacier to S3).

I have done some research and didn't find any possible way using Amazon's API.

PROBLEM

How can I move files between S3 and glacier using API?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Irfan DANISH
  • 8,349
  • 12
  • 42
  • 67
  • Did you take a look at APIs for S3 Lifecycle Configuration? I have done it using Python boto. Not sure about PHP. – helloV Nov 24 '14 at 07:53
  • 3
    This question gives the impression that what you are in need of, first, is a more thorough understanding of how S3's Glacier integration actually works at cconceptual level... manually migrating to Glacier is not a thing, and when files are restored to S3 from Glacier, that's temporary; they are also still stored in Glacier, not moved back to S3. – Michael - sqlbot Nov 24 '14 at 16:04

3 Answers3

33

You could use the Glacier API to upload a file to a Glacier vault, but I don't recommend it. The previous version of our backup app did that. When you upload a file it gets a randomly-assigned name. You can add put your filename in the metadata of the file, but if you want a list of what's in the Glacier vault you have to query and then wait 3-5 hours for the list.

Lifecycle policies are the other way to use Glacier. The current version of Arq uses them because each object still looks like an S3 object (no random object names, no delays in getting object lists), but the object contents are in Glacier storage. The only difference is that getting the object contents is a 2-step process: you have to make an API call to request that the object be made downloadable; when it's ready, you can download it. Also there's a "peak hourly request fee" that comes into play if you request objects be made downloadable at too fast a rate. Amazon Glacier pricing is complex.

Once an object is "Glacier storage class" there's no way to change it back to "Standard storage class". You have to make a copy of the object that's "Standard storage class" and delete the Glacier object.

So maybe a simple solution to your problem is:

  1. Store the data in 2 "folders" in S3, "standard" and "glacier".
  2. Set a lifecycle policy to push all objects in the "glacier" folder to Glacier data storage ASAP.
  3. When you want to move an object from standard to glacier, copy it to the glacier folder and delete the object in the standard folder (there's no "move" API).
  4. When you want to move an object from glacier to standard, do a POST request to restore it; when it's restored, copy it to the standard folder and delete it from the glacier folder.
Stefan
  • 1,248
  • 11
  • 14
  • If there's a purpose and significance to the files's current location (e.g. a redshift dump with a manifest of hardcoded s3 locations), then can create a lifecycle policy pointing to a prefix and if you don't want everything under that prefix, you can specify tags for the lifecycle policy too. – combinatorist Jan 23 '20 at 20:50
7

You can use the API to define lifecycle rules that archive files from Amazon S3 to Amazon Glacier and you can use the API to retrieve a temporary copy of files archived to Glacier. However, you cannot use the API to tell Amazon S3 to move specific files into Glacier.

There are two ways to use Amazon Glacier:

  1. Directly via the Glacier API, which allows you to upload/download archives to/from Glacier vaults
  2. Via Amazon S3 lifecycle rules, which archive data from Amazon S3 into Amazon Glacier

Connecting directly via the Glacier API allows you to store archives for long-term storage, often used as a replacement for Tape. Data stored via the Glacier API must also be retrieved via the Glacier API. This is typically done with normal enterprise backup software or even light-weight products such as Cloudberry Backup (Windows) or Arq (Mac).

Using Amazon S3 lifecycle rules allows you to store data in Amazon S3, then define rules that determine when data should be archived to Glacier for long-term storage. For example, data could be archived 90 days after creation. The data transfer is governed by the lifecycle rules, which operate on a daily batch basis. The rules can be set via the putBucketLifecycle API call (available in the PHP SDK), but this only defines the rules -- it is not possible to make an API call that tells S3 to archive specific files to Glacier.

Amazon S3 has a RestoreObject API call (available in the PHP SDK) to restore a temporary copy of data archived from Glacier back into S3. Please note that restoring data from Glacier takes 3-5 hours.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
1

To Move files from Glacier to S3 standard:

Using S3 Browser:

Step1: click the object and select restore from Glacier this will make the object available.(Give the number of days you want the object to needed in Standard class)

Step2: Again and Select the option "Change Storage Class" this will change the storage class of the object.

Using CLI:

Step1: aws s3api restore-object --bucket awsexamplebucket --key dir1/example.obj --restore-request '{"Days":25,"GlacierJobParameters":{"Tier":"Standard"}}'this makes the object to be available based on the storage class 4 hours for glacier.

Step2: run this command in CLI:

After the object is available run this command to change the object to Standard class aws s3 cp s3://<object path> s3://<destination bucket path> --storage-class STANDARD --recursive --force-glacier-transfer

followed this link -[1]: https://aws.amazon.com/premiumsupport/knowledge-center/restore-s3-object-glacier-storage-class/

Simba
  • 37
  • 6