How to change metadata on an object in Amazon S3

Question

If you have already uploaded an object to an Amazon S3 bucket, how do you change the metadata using the API? It is possible to do this in the AWS Management Console, but it is not clear how it could be done programmatically. Specifically, I'm using the boto API in Python and from reading the source it is clear that using key.set_metadata only works before the object is created as it just effects a local dictionary.

score 37 · Accepted Answer · edited May 23 '17 at 11:46

37

It appears you need to overwrite the object with itself, using a "PUT Object (Copy)" with an x-amz-metadata-directive: REPLACE header in addition to the metadata. In boto, this can be done like this:

k = k.copy(k.bucket.name, k.name, {'myKey':'myValue'}, preserve_acl=True)

Note that any metadata you do not include in the old dictionary will be dropped. So to preserve old attributes you'll need to do something like:

k.metadata.update({'myKey':'myValue'})
k2 = k.copy(k.bucket.name, k.name, k.metadata, preserve_acl=True)
k2.metadata = k.metadata    # boto gives back an object without *any* metadata
k = k2;

I almost missed this solution, which is hinted at in the intro to an incorrectly-titled question that's actually about a different problem than this question: Change Content-Disposition of existing S3 object

edited May 23 '17 at 11:46

Community

1
1

answered Jan 21 '11 at 01:07

natevw

16,807
8
66
90

2

I don't see in the second code example where the union between old and new metadata is taking place. When I execute the code, the metadata for the object gets overwritten with the latest metadata. – John Carter May 26 '11 at 16:51
1

Seems to work, but the new key loses the Content-Type, which is really bad when using S3 to serve web images. – Ryan_IRL Feb 03 '12 at 21:43
1

For those of you using the AWS SDK you need to use [AmazonS3Client.copyObject method](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#copyObject(com.amazonaws.services.s3.model.CopyObjectRequest)) – glidester Jun 20 '13 at 10:12
Does it copying the content of the key or only the metadata? Copying the whole key takes time, and decrease the efficiency, right? – Celik Aug 08 '14 at 12:46
This also seems to replace the infamous multipart ETag with a regular MD5 of the entire file - which kills two birds with one stone for me. The boto3 package does not seem to do this incidentally. In summary: GREAT +1 :) – Darragh Enright Jan 12 '16 at 10:39
Actually - I stand corrected - boto3's s3 package has `client.copy_object` which creates an MD5 ETag - at least with the 150MB objects I tested. However, you cannot "copy over" an existing object, unlike `Key.copy` above. – Darragh Enright Jan 12 '16 at 14:09

kartik · Answer 2 · 2015-04-26T19:02:00.417

12

In order to set metadata on S3 files,just don't provide target location as only source information is enough to set metadata.

final ObjectMetadata metadata = new ObjectMetadata();
metadata.addUserMetadata(metadataKey, value);
final CopyObjectRequest request = new CopyObjectRequest(bucketName, keyName, bucketName, keyName)
  .withSourceBucketName(bucketName)
  .withSourceKey(keyName)
  .withNewObjectMetadata(metadata);

s3.copyObject(request);`

edited Apr 26 '15 at 19:02

answered May 14 '13 at 10:37

kartik

2,097
3
21
31

2

The question asked for Python, but this was the only answer using the Java SDK that I've been able to find on SO. Thanks! – Shotgun Ninja May 01 '14 at 16:18
We are able to put tag on file of any size but not sure, if metadata on heavy file issue is fixed or not. – kartik May 01 '17 at 09:53

score 8 · Answer 3 · answered Apr 14 '15 at 10:35

8

If you want your metadata stored remotely use set_remote_metadata

Example: key.set_remote_metadata({'to_be': 'added'}, ['key', 'to', 'delete'], {True/False})

Implementation is here: https://github.com/boto/boto/blob/66b360449812d857b4ec6a9834a752825e1e7603/boto/s3/key.py#L1875

answered Apr 14 '15 at 10:35

eddd

1,715
1
11
10

This is the correct answer, and the easiest way to do what @natevw's answer describes. It does the same thing under the hood. Note that `metadata_minus` and `preserve_acl` are required. – nathancahill Nov 11 '15 at 21:33
key.set_metadata() works too (though it just replace the existing metadata). set_remote_metadata() supports addition and deleting with existing metadata. This should be the correct answer, compare with the current top answer. – berniey Nov 17 '17 at 23:19
1

A note for anyone who comes here looking nowadays: this is no longer possible in boto3. Be sure to refine your search for boto3. – Oded Nov 09 '20 at 21:57

score 5 · Answer 4 · edited May 23 '17 at 12:34

5

You can change the metadata without re-uloading the object by using the copy command. See this question: Is it possible to change headers on an S3 object without downloading the entire object?

edited May 23 '17 at 12:34

Community

1
1

answered Feb 18 '12 at 02:18

John Bachir

22,495
29
154
227

score 1 · Answer 5 · edited May 17 '13 at 21:44

1

For the first answer it's a good idea to include the original content type in the metadata, for example:

key.set_metadata('Content-Type', key.content_type)

edited May 17 '13 at 21:44

samayo

16,163
12
91
106

answered May 17 '13 at 21:23

Ted

592
6
8

Been a while since I've used it, but I'm pretty sure that's among the things passing the original `k.metadata` field on the copy accomplishes. – natevw May 24 '13 at 20:12

score -1 · Answer 6 · answered Apr 23 '15 at 05:23

In Java, You can copy object to the same location. Here metadata will not copy while copying an Object. You have to get metadata of original and set to copy request. This method is more recommended to insert or update metadata of an Amazon S3 object

ObjectMetadata metadata = amazonS3Client.getObjectMetadata(bucketName, fileKey);
ObjectMetadata metadataCopy = new ObjectMetadata();
metadataCopy.addUserMetadata("yourKey", "updateValue");
metadataCopy.addUserMetadata("otherKey", "newValue");
metadataCopy.addUserMetadata("existingKey", metadata.getUserMetaDataOf("existingValue"));

CopyObjectRequest request = new CopyObjectRequest(bucketName, fileKey, bucketName, fileKey)
      .withSourceBucketName(bucketName)
      .withSourceKey(fileKey)
      .withNewObjectMetadata(metadataCopy);

amazonS3Client.copyObject(request);

score -1 · Answer 7 · answered Sep 04 '15 at 05:35

here is the code that worked for me. I'm using aws-java-sdk-s3 version 1.10.15

ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType(fileExtension.getMediaType());

s3Client.putObject(new PutObjectRequest(bucketName, keyName, tempFile)
                    .withMetadata(metadata));

How to change metadata on an object in Amazon S3

7 Answers7

Linked