Multiple file upload DRF

Question

I have a requirement which I would like to allow multiple files to be uploaded within the same post request to create an object. I currently have a method of doing this, but after looking at some other examples it doesn't appear to be intended way to do it.

models.py

class Analyzer(models.Model):
    name = models.CharField(max_length=100, editable=False, unique=True)

class Atomic(models.Model):
    name = models.CharField(max_length=20, unique=True)

class Submission(models.Model):
    class Meta:
        ordering = ['-updated_at']

    issued_at = models.DateTimeField(auto_now_add=True, editable=False)
    completed = models.BooleanField(default=False)
    analyzers = models.ManyToManyField(Analyzer, related_name='submissions')
    atomic = models.ForeignKey(Atomic, verbose_name='Atomic datatype', related_name='submission', on_delete=models.CASCADE)

class BinaryFile(models.Model):
    class Meta:
        verbose_name = 'Binary file'
        verbose_name_plural = 'Binary files'
    def __str__(self):
        return self.file.name

    submission = models.ForeignKey(Submission, on_delete=models.CASCADE, related_name='binary_files')
    file = models.FileField(upload_to='uploads/binary/')

serializers.py

class BinaryFileSerializer(serializers.ModelSerializer):
    class Meta:
        model = models.BinaryFile
        fields = '__all__'

class SubmissionCreateSerializer(serializers.ModelSerializer):
    class Meta:
        model = models.Submission
        fields = ['id', 'completed', 'atomic', 'analyzers', 'binary_files']

    id = serializers.ReadOnlyField()
    completed = serializers.ReadOnlyField()

    atomic = serializers.PrimaryKeyRelatedField(many=False, queryset=models.Atomic.objects.all()
    analyzers = serializers.PrimaryKeyRelatedField(many=True, queryset=models.Analyzer.objects.all()

    binary_files = BinaryFileSerializer(required=True, many=True)

    def validate(self, data):
        # # I dont really like manually taking invalidated input!!
        data['binary_files'] = self.initial_data.getlist('binary_files')
        return data

    def create(self, validated_data):

        submission = models.Submission.objects.create(
            atomic=validated_data['atomic']
        )
        submission.analyzers.set(validated_data['analyzers'])

        # # Serialize the files - this seems too late to be doing this!
        for file in validated_data['binary_files']:
            binary_file = BinaryFileSerializer(
                data={'file': file, 'submission': submission.id}
            )

            if binary_file.is_valid():
                binary_file.save()

        return submission

Main question: While the above works, the child serializer (BinaryFileSerializer) doesn't get called until I explicitly call it in create(), which is after the validation should have occurred. Why does this never get called?

I also don't like the fact I have to manually do a self.initial_data.getlist('binary_files') and manually add it to data - this should have already been added and validated, no?

My thought is that as I defined binary_files = BinaryFileSerializer, this serializer should be called to validate that particular fields input?

FYI, I'm using the following to test POST uploads:

curl -F "binary_files=@file2.txt" -F "binary_files=@file2.txt" -F "atomic=7" -F "analyzers=12" -H "Accept: application/json; indent=4"  http://127.0.0.1:8000/api/submit/

TIA!

Update: The question is now, if a validate() funciton is added to the BinaryFileSerializer, why does it not get called?

Answer 1

Possible duplicate --- Django REST: Uploading and serializing multiple images.

---

---

From the DRF Writable Nested Serializer doc,

By default nested serializers are read-only. If you want to support write-operations to a nested serializer field you'll need to create create() and/or update() methods in order to explicitly specify how the child relationships should be saved.

From this, it's clear that the child serializer (BinaryFileSerializer) won't call its own create() method unless explicitly called.

The aim of your HTTP POST request is to create new Submission instance (and BinaryFile instance). The creation process undergoes in the create() method of the SubmissionCreateSerializer serializer, which is you'd overridden. So, it will act/execute as per your code.

---

UPDATE-1

Things to remember
1. AFAIK, we can't send nested multipart/form-data
2. Here I'm only trying to implementing the least case scenario
3. I'm tested this solution with POSTMAN rest api test tool.
4. This method may be complex (until we found a better one).
5. Assuming your view class is subclass of ModelViewSet class


What I'm going to do?
1. Since we can't send the files/data in a nested fashion, we have to send it flat mode.

image-1


2. Override the __init__() method of the SubmissionSerializer serializer and dynamically add as much FileField() attribute according to the request.FILES data.
We could somehow use ListSerializer or ListField here. Unfortunately I couldn't find out a way :(

# init method of "SubmissionSerializer"
def __init__(self, *args, **kwargs):
    file_fields = kwargs.pop('file_fields', None)
    super().__init__(*args, **kwargs)
    if file_fields:
        field_update_dict = {field: serializers.FileField(required=False, write_only=True) for field in file_fields}
        self.fields.update(**field_update_dict)

So, what id file_fields here?
Since the form-data is a key-value pair, every file data must be associated with a key. Here in image-1, you could see file_1 and file_2.

3. Now we need to pass the file_fields values from the view. Since this operation is creating new instance, we need to override the create() method of the API class.

# complete view code
from rest_framework import status
from rest_framework import viewsets


class SubmissionAPI(viewsets.ModelViewSet):
    queryset = Submission.objects.all()
    serializer_class = SubmissionSerializer

    def create(self, request, *args, **kwargs):
        # main thing starts
        file_fields = list(request.FILES.keys())  # list to be passed to the serializer
        serializer = self.get_serializer(data=request.data, file_fields=file_fields)
        # main thing ends

        serializer.is_valid(raise_exception=True)
        self.perform_create(serializer)
        headers = self.get_success_headers(serializer.data)
        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)

4. Now, all values will be serialized properly. It's time to override the create() method of the SubmissionSerializer() to map the relations

def create(self, validated_data):
    from django.core.files.uploadedfile import InMemoryUploadedFile
    validated_data_copy = validated_data.copy()
    validated_files = []
    for key, value in validated_data_copy.items():
        if isinstance(value, InMemoryUploadedFile):
            validated_files.append(value)
            validated_data.pop(key)
    submission_instance = super().create(validated_data)
    for file in validated_files:
        BinaryFile.objects.create(submission=submission_instance, file=file)
    return submission_instance

5. That's it!!!

---

Complete Code Snippet

# serializers.py
from rest_framework import serializers
from django.core.files.uploadedfile import InMemoryUploadedFile


class SubmissionSerializer(serializers.ModelSerializer):
    def __init__(self, *args, **kwargs):
        file_fields = kwargs.pop('file_fields', None)
        super().__init__(*args, **kwargs)
        if file_fields:
            field_update_dict = {field: serializers.FileField(required=False, write_only=True) for field in file_fields}
            self.fields.update(**field_update_dict)

    def create(self, validated_data):
        validated_data_copy = validated_data.copy()
        validated_files = []
        for key, value in validated_data_copy.items():
            if isinstance(value, InMemoryUploadedFile):
                validated_files.append(value)
                validated_data.pop(key)
        submission_instance = super().create(validated_data)
        for file in validated_files:
            BinaryFile.objects.create(submission=submission_instance, file=file)
        return submission_instance

    class Meta:
        model = Submission
        fields = '__all__'


# views.py
from rest_framework import status
from rest_framework import viewsets


class SubmissionAPI(viewsets.ModelViewSet):
    queryset = Submission.objects.all()
    serializer_class = SubmissionSerializer

    def create(self, request, *args, **kwargs):
        # main thing starts
        file_fields = list(request.FILES.keys())  # list to be passed to the serializer
        serializer = self.get_serializer(data=request.data, file_fields=file_fields)
        # main thing ends

        serializer.is_valid(raise_exception=True)
        self.perform_create(serializer)
        headers = self.get_success_headers(serializer.data)
        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)

---

Screenhots and other stuffs

1. POSTMAN console

2. Django Shell

In [2]: Submission.objects.all()                                                                                                                                                                                   
Out[2]: <QuerySet [<Submission: Submission object>]>

In [3]: sub_obj = Submission.objects.all()[0]                                                                                                                                                                      

In [4]: sub_obj                                                                                                                                                                                                    
Out[4]: <Submission: Submission object>

In [5]: sub_obj.__dict__                                                                                                                                                                                           
Out[5]: 
{'_state': <django.db.models.base.ModelState at 0x7f529a7ea240>,
 'id': 5,
 'issued_at': datetime.datetime(2019, 3, 27, 8, 45, 42, 193943, tzinfo=<UTC>),
 'completed': False,
 'atomic_id': 1}

In [6]: sub_obj.binary_files.all()                                                                                                                                                                                 
Out[6]: <QuerySet [<BinaryFile: uploads/binary/logo-800.png>, <BinaryFile: uploads/binary/Doc.pdf>, <BinaryFile: uploads/binary/invoice_2018_11_29_04_57_53.pdf>, <BinaryFile: uploads/binary/Screenshot_from_2019-02-13_16-22-53.png>]>

In [7]: for _ in sub_obj.binary_files.all(): 
   ...:     print(_) 
   ...:                                                                                                                                                                                                            
uploads/binary/logo-800.png
uploads/binary/Doc.pdf
uploads/binary/invoice_2018_11_29_04_57_53.pdf
uploads/binary/Screenshot_from_2019-02-13_16-22-53.png

3. Django Admin Screenhot