How do you add user-defined S3 metadata in when using django-storages

Question

I am using django 3.1.0 with django-storages 1.12.3. I am using an S3 backend to store media files (Minio). I am trying to post the file to S3, and include additional user metadata to the S3 object. This additional metadata comes from the POST request used to upload the file to django. With S3, custom metadata can be added by including additional key-value pairs to the HTTP header for the POST request sent to the S3 server when the file is uploaded, where the custom keys would be prefixed with 'X-Amz-Meta-'.

I am using a FileField in a django model, and files are uploaded using a REST API endpoint. As far as I understand it, when django receives the file, it stores it temporarily, and then, when saving the FielField on the model instance, the file is posted to the S3 server. I want to modify the flow, so that the custom metadata is taken from the request and included in the post to the S3 server.

Any idea how I can take data from the initial request, and pass it to the header of the POST request being sent to S3?

Update

After trying Option 1 from Helge Schneider's answer, I did a little modification and got it to work.

I am using django rest framework, so I modified the serializer .save() method.

from .models import MyModel
from rest_framework import serializers

class MyModelSerialzer(serializers.ModelSerializer):

    class Meta:
        model = SessionFile
        fields = ['file', 'my_other_field']

    def save(self):
        file = self.validated_data['file']
        my_other_field = self.validated_data['my_other_field']
        mymodel = MyModel(file=file, my_other_field=my_other_field)
        options_dict = {"Metadata": {"metadata1": "ImageName", 
                                    "metadata2": "ImagePROPERTIES",
                                    "metadata3": "ImageCREATIONDATE"}
                        } 
        mymodel.file.storage.object_parameters.update(options_dict)
        mymodel.save()

Helge Schneider · Accepted Answer · 2022-03-06T16:20:53.690

1

There are two possible options:

1. Update the parameters of the storage class before calling the save method

One option would be to update the storage class from the filefield of the model before calling the save method and update the object_parameters attribute.

views.py

import MyModel


def my_view(request):
    mymodel = MyModel()
    options_dict = {"Metadata": {"metadata1": "ImageName", 
                                 "metadata2": "ImagePROPERTIES",
                                 "metadata3": "ImageCREATIONDATE"}
                    } 
    mymodel.filefield.storage.object_parameters.update(options_dict)
    mymodel.filefield.save()

2. Manually upload the file with boto3

Another option would be to use a boto3 client to upload the file manually (Documentation with the upload_fileobj method.

You can then set the object metadata with the following code taken from this answer:

import boto3
s3 = boto3.resource('s3')
options_dict = {"Metadata": {"metadata1": "ImageName", 
                             "metadata2": "ImagePROPERTIES",
                             "metadata3": "ImageCREATIONDATE"}
s3.upload_fileobj(file, bucketname, key, ExtraArgs=options_dict)

Afterwards you have to set the name property of the FileField to the key in the S3 Bucket as described in this answer.

object.filefield.name = 'file/key/in/bucket'

edited Mar 06 '22 at 16:20

answered Mar 03 '22 at 16:26

Helge Schneider

483
5
8

I think there would still be a problem with the first option. The issue is that `get_object_parameters` receives as arguments `self` and `name`. `name` is just a string, and `self` is an instance of `CustomStorage`, which inherits from `storages.S3Boto3Storage`, which inherits from `django.core.files.Storage`. None of the storage classes receive `request`, so I cannot see how I would access the request inside the `get_object_parameters` method. – MG79 Mar 04 '22 at 16:38
For the second option, I suppose I would need to also write a custom upload handler. If I would override the model's `save` method, so that the file is posted to S3, then I would not be able to use the `boto3.client.upload_file` method because it takes Filename as an argument. During `django's model.save`, the file is not necessarily stored on the local file system. For smaller file sizes, it would be stored in memory. [link] (https://docs.djangoproject.com/en/4.0/ref/files/uploads/#django.core.files.uploadhandler.TemporaryFileUploadHandler). I guess I could use `boto3.client.upload_fileobj`. – MG79 Mar 04 '22 at 17:24
You are right, with the original first answer it was not possible to pass the request to the storage class. I updated the first option by trying to access the object_parameters from the filefield directly before calling the save method. Can you please check if this approach works? – Helge Schneider Mar 04 '22 at 18:46
For the second option, you would do the boto call in the view before saving the filefield, set the `name` attribute on the filefield and then save the filefield. You do not neccessarily have to overwrite the save method of the model. – Helge Schneider Mar 04 '22 at 18:47
Hi Helge, as you suggested, adding the options_dict worked. I made an edit to my question to show how it was implemented. Your second option also worked, but the boto3 method `.upload_file()` needed to be changed to `.upload_fileobj` – MG79 Mar 06 '22 at 15:35
Great that it worked! I edited the answer to use the `.upload_fileobj` method. – Helge Schneider Mar 06 '22 at 16:21

How do you add user-defined S3 metadata in when using django-storages

Update

1 Answers1