Setting metadata on S3 multipart upload

Question

I'd like to upload a file to S3 in parts, and set some metadata on the file. I'm using boto to interact with S3. I'm able to set metadata with single-operation uploads like so:

Is there a way to set metadata with a multipart upload? I've tried this method of copying the key to change the metadata, but it fails with the error: InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: <size>

I've also tried doing the following:

key = bucket.create_key(key_name)
key.set_metadata('some-key', 'value')
<multipart upload>

...but the multipart upload overwrites the metadata.

I'm using code similar to this to do the multipart upload.

score 5 · Accepted Answer · answered Dec 05 '14 at 22:54

5

Sorry, I just found the answer:

Per the docs:

If you want to provide any metadata describing the object being uploaded, you must provide it in the request to initiate multipart upload.

So in boto, the metadata can be set in the initiate_multipart_upload call. Docs here.

answered Dec 05 '14 at 22:54

user3776364

153
1
8

How can I use SSE-KMS along with multipart upload? Complete question is here: https://stackoverflow.com/questions/47328843/s3-multipart-upload-with-sse-kms – Vikash Pareek Nov 17 '17 at 07:09

score 2 · Answer 2 · answered Sep 23 '16 at 15:56

Faced such issue earlier today and discovered that there is no information on how to do that right. The code example on how we solved that issue provided below.

        $uploader = new MultipartUploader($client, $source, [
        'bucket' => $bucketName,
        'key' => $filename,
        'before_initiate' => function (\Aws\Command $command) {
            $command['ContentType'] = 'application/octet-stream';
            $command['ContentDisposition'] = 'attachment';
        },
    ]);

Unfortunately, documentation https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html#customizing-a-multipart-upload doesn't make it clear and easy to understand that if you'd like to provide alternative meta data with multipart upload you have to go this way.

I hope that will help.

Setting metadata on S3 multipart upload

2 Answers2