How do I update metadata for an existing Amazon S3 file?

Question

I need to update the cache control header in all AmazonS3's Cloud Files. However, I can't figure out how I do that using the jclouds API. I'm using apache jclouds plugin. And I got two related answers:

The first answer is suggesting to use SwiftKey Api class which is not available in grails's jcloud plugin. The second answer is using AWS java sdk for which there is already a grails wrapping plugin https://grails.org/plugin/aws-sdk but it doesn't support metadata update.

score 67 · Accepted Answer · edited May 23 '17 at 12:26

67

It is possible to change the metadata by performing an object copy (see How to update metadata using Amazon S3 SDK):

ObjectMetadata metadataCopy = new ObjectMetadata();
// copy previous metadata
metadataCopy.addUserMetadata("newmetadata", "newmetadatavalue");

CopyObjectRequest request = new CopyObjectRequest(bucketName, existingKey, bucketName, existingKey)
      .withNewObjectMetadata(metadataCopy);

amazonS3Client.copyObject(request);

Whether this is philosophically an "update" is up to you to decide.

edited May 23 '17 at 12:26

Community

1
1

answered Aug 11 '16 at 11:03

Dan Gravell

7,855
7
41
64

1

Thanks Dan, I end up doing the same thing. – Priyanshu Chauhan Oct 18 '16 at 09:38

score 36 · Answer 2 · answered Sep 18 '15 at 08:00

36

You can't:

Each Amazon S3 object has data, a key, and metadata. Object key (or key name) uniquely identifies the object in a bucket. Object metadata is a set of name-value pairs. You can set object metadata at the time you upload it. After you upload the object, you cannot modify object metadata. The only way to modify object metadata is to make a copy of the object and set the metadata.

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

answered Sep 18 '15 at 08:00

E.J. Brennan

45,870
7
88
116

3

I find the AWS documentation on the metadata topic highly confusing - Along side the link you've provided, I found another link to their User Guide, where it shows how to add or delete object metadata using the management console. The only resolution to this conflict that I can come up with, is that when they say "you cannot modify object metadata", they mean, you cannot modify the value of a specific meta data key. what's your thought? http://docs.aws.amazon.com/AmazonS3/latest/UG/EditingtheMetadataofanObject.html – Israel Aug 16 '16 at 08:12
Perhaps they mean you cannot modify the metadata via the REST interface but you can via the management console? (Behind the scenes it would probably clone and delete). – kaifong Mar 10 '17 at 16:28
See my solution below, you can update an existing S3 object and setting the new/updated metadata at the same time. It's similar to the copy solution. – robnick Mar 01 '18 at 04:06
I realize this is an old thread but the documentation on ["Editing object metadata in the Amazon S3 console"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-object-metadata.html) now clearly states that "This action creates a copy of the object with updated settings and the last-modified date." So no special treatment for the web console. – Giorgio Ruffa Apr 19 '21 at 15:21

score 7 · Answer 3 · answered Jun 25 '19 at 15:57

PHP Example

I understand this question was not PHP specific, but it may help someone as it is a top result on Google.

This will overwrite the existing object.

$client = new \Aws\S3\S3Client([
    'version' => '2006-03-01',
    'region' => 'BUCKET-REGION'
]);

$updateResponse = $client->copyObject([
    'Key' => 'OBJECTKEY',
    'Bucket' =>  'MYBUCKET',
    'CopySource' => 'MYBUCKET/OBJECTKEY',
    'MetadataDirective' => 'REPLACE',
    'Metadata' => [
        'width' => 441,
        'height' => 189
    ]
]);

Can you help me update the metadata of the existing object in laravel PHP.I have tried your solution but unable to update the metadata of the existing file.Thanks — Bhargav Variya, May 20 '21 at 11:42

JellicleCat · Answer 4 · 2021-02-17T01:24:05.317

3

As stated in other answers, a "copy" request can be made which overwrites the existing record.

Ruby example

s3_client.copy_object({
  bucket: BUCKET, # Destination
  key: KEY, # Destination
  copy_source: "/#{BUCKET}/#{KEY}", # Source
  content_type: "text/html; charset=utf8", # Metadata update
  metadata_directive: "REPLACE"
})

edited Feb 17 '21 at 01:24

answered Feb 17 '21 at 01:02

JellicleCat

28,480
24
109
162

1

Thanks. Side note: Funny how in java/ruby world this old question is still relevant ! The OP works with javascript frameworks, where even a 2 year old question becomes irrelevant. – Priyanshu Chauhan Feb 24 '21 at 08:22
1

@PriyanshuChauhan it shows how immature the JS world is ;) – GuyT Jun 23 '21 at 06:41

score 1 · Answer 5 · answered Oct 19 '22 at 15:48

I had to do it recently and used JS for that:

const {
  S3Client,
  ListObjectsCommand,
  HeadObjectCommand,
  CopyObjectCommand,
} = require('@aws-sdk/client-s3');
const fs = require('fs');

const client = new S3Client({
  region: 'us-west-2',
});

const files = fs.readFileSync('images.txt', 'utf8').split('\n');

async function run() {
  for (const prefix of files) {
    await fixMetdata(prefix);
  }
}

const BUCKET_NAME = 'my-bucket';

async function fixMetdata(prefix) {
  // console.error(prefix);
  try {
    const listObjects = new ListObjectsCommand({
      Bucket: BUCKET_NAME,
      Prefix: prefix,
      MaxKeys: 1,
    });
    const data = await client.send(listObjects);

    if (!data.Contents) return;

    data.Contents.filter((f) => f.Key.includes('.jpg')).forEach(
      async (file) => {
        const fileMetadata = await client.send(
          new HeadObjectCommand({ Bucket: BUCKET_NAME, Key: file.Key })
        );
        // console.error(fileMetadata.ContentType);
        if (fileMetadata.ContentType !== 'image/jpeg') {
          const updatedMetadata = await client.send(
            new CopyObjectCommand({
              Bucket: BUCKET_NAME,
              Key: file.Key,
              CopySource: `${BUCKET_NAME}/${file.Key}`,
              MetadataDirective: 'REPLACE',
              ContentType: 'image/jpeg',
            })
          );
          console.log(file.Key);
        }
      }
    );
  } catch (err) {
    console.error('Error', err);
  }
}

run();

score 0 · Answer 6 · answered Dec 13 '19 at 17:10

0

In my case, I care about the object versions, and copying creates a new version. I decided to create a parallel, "hidden" object with the mutable metadata. For example, if I have /foo/bar/baz, then I would create /foo/bar/.baz.

answered Dec 13 '19 at 17:10

John Lindal

616
5
11

So for each file you upload to AWS S3, you uploaded a second hidden file with the MD5 embedded in it? Doesn't that increase the costs of your PUTs and GETs? – Edward Coast Jan 17 '20 at 12:47
It does, but my application isn't high volume. – John Lindal Jan 17 '20 at 20:15

score 0 · Answer 7 · answered Apr 25 '23 at 14:41

I couldn't find it documented anywhere, but if you are using Step Functions to make an SDK call to S3, you can set metadata like this, using MetadataDirective like with PHP and Ruby above (this example shows the parameters for s3:copyObject):

{
  "Bucket.$": "$.bucketName",
  "CopySource.$": "$.bucketSourceLocation",
  "Key.$": "$.destinationKey",
  "Acl": "bucket-owner-full-control",
  "ServerSideEncryption": "AES256",
  "MetadataDirective": "REPLACE",
  "Metadata": {
    "my-metadata.$": "$.my-metadata"
  }
}

score -3 · Answer 8 · answered Mar 01 '18 at 04:04

As at March 2018, the latest version of the AWS SDK for .NET Core has changed. It now uses asynchronous programming. Many of the method signatures have changed. While you still cannot change metadata without the object copy solution that Dan has suggested, the code to do so has.

My solution is to update the existing S3 object with the modified metadata.

The following works for me to update a single metadata value (based on key and new value). I've got two loops for setting the metadata but it could be optimised to just have one:

string fileContents = string.Empty;
Dictionary<string, string> fileMetaData = null;

GetObjectRequest request = new GetObjectRequest
{
  BucketName = bucketName,
  Key = setKeyName
};

var response = await s3Client.GetObjectAsync(request);

// Read the contents of the file
using (var stream = response.ResponseStream)
{
  // Get file contents
  TextReader tr = new StreamReader(stream);
  fileContents = tr.ReadToEnd();
}

// Create the File Metadata collection
fileMetaData = new Dictionary<string, string>();
foreach (string metadataKey in response.Metadata.Keys)
{
  fileMetaData.Add(metadataKey, response.Metadata[metadataKey]);
}

// Update the metadata value (key to update, new value)
fileMetaData[metaDataKeyToUpdate] = metaDataNewValue;

// Update the existing S3 object    
PutObjectRequest putRequest1 = new PutObjectRequest
{
  BucketName = bucketName,
  Key = setKeyName,
  ContentBody = fileContents
};

// Set the metadata
foreach (string metadataKey in response.Metadata.Keys)
{
  putRequest1.Metadata.Add(metadataKey, fileMetaData[metadataKey]);
}

PutObjectResponse response1 = await s3Client.PutObjectAsync(putRequest1);

That downloads entire file on the client, in memory, so works only for very limited use cases. — Ivan G., Aug 20 '19 at 07:20
Yep you are right. Guilty as charged! PST: Object Copy is nasty as all heck :) — robnick, Aug 21 '19 at 10:21

How do I update metadata for an existing Amazon S3 file?

8 Answers8

Ruby example

Linked