How can I test binary file uploading with django-rest-framework's test client?

Question

I have a Django application with a view that accepts a file to be uploaded. Using the Django REST framework I'm subclassing APIView and implementing the post() method like this:

class FileUpload(APIView):
    permission_classes = (IsAuthenticated,)

    def post(self, request, *args, **kwargs):
        try:
            image = request.FILES['image']
            # Image processing here.
            return Response(status=status.HTTP_201_CREATED)
        except KeyError:
            return Response(status=status.HTTP_400_BAD_REQUEST, data={'detail' : 'Expected image.'})

Now I'm trying to write a couple of unittests to ensure authentication is required and that an uploaded file is actually processed.

class TestFileUpload(APITestCase):
    def test_that_authentication_is_required(self):
        self.assertEqual(self.client.post('my_url').status_code, status.HTTP_401_UNAUTHORIZED)

    def test_file_is_accepted(self):
        self.client.force_authenticate(self.user)
        image = Image.new('RGB', (100, 100))
        tmp_file = tempfile.NamedTemporaryFile(suffix='.jpg')
        image.save(tmp_file)
        with open(tmp_file.name, 'rb') as data:
            response = self.client.post('my_url', {'image': data}, format='multipart')
            self.assertEqual(status.HTTP_201_CREATED, response.status_code)

But this fails when the REST framework attempts to encode the request

Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/django/utils/encoding.py", line 104, in force_text
    s = six.text_type(s, encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 118: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/webapp/myproject/myapp/tests.py", line 31, in test_that_jpeg_image_is_accepted
    response = self.client.post('my_url', { 'image': data}, format='multipart')
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-    packages/rest_framework/test.py", line 76, in post
    return self.generic('POST', path, data, content_type, **extra)
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/rest_framework/compat.py", line 470, in generic
    data = force_bytes_or_smart_bytes(data, settings.DEFAULT_CHARSET)
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/django/utils/encoding.py", line 73, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/django/utils/encoding.py", line 116, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)
django.utils.encoding.DjangoUnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 118: invalid start byte. You passed in b'--BoUnDaRyStRiNg\r\nContent-Disposition: form-data; name="image"; filename="tmpyz2wac.jpg"\r\nContent-Type: image/jpeg\r\n\r\n\xff\xd8\xff[binary data omitted]' (<class 'bytes'>)

How can I make the test client send the data without attempting to decode it as UTF-8?

@arocks Eagle eyes! I've corrected the typo in the posting, the actual code did not have this problem. — Tore Olsen, Jun 13 '14 at 09:26

score 58 · Answer 1 · edited Jul 24 '19 at 19:25

58

When testing file uploads, you should pass the stream object into the request, not the data.

This was pointed out in the comments by @arocks

Pass { 'image': file} instead

But that didn't full explain why it was needed (and also didn't match the question). For this specific question, you should be doing

from PIL import Image

class TestFileUpload(APITestCase):

    def test_file_is_accepted(self):
        self.client.force_authenticate(self.user)

        image = Image.new('RGB', (100, 100))

        tmp_file = tempfile.NamedTemporaryFile(suffix='.jpg')
        image.save(tmp_file)
        tmp_file.seek(0)

        response = self.client.post('my_url', {'image': tmp_file}, format='multipart')

       self.assertEqual(status.HTTP_201_CREATED, response.status_code)

This will match a standard Django request, where the file is passed in as a stream object, and Django REST Framework handles it. When you just pass in the file data, Django and Django REST Framework interpret it as a string, which causes issues because it is expecting a stream.

And for those coming here looking to another common error, why file uploads just won't work but normal form data will: make sure to set format="multipart" when creating the request.

This also gives a similar issue, and was pointed out by @RobinElvin in the comments

It was because I was missing format='multipart'

edited Jul 24 '19 at 19:25

Julio Marins

10,039
8
48
54

answered Dec 20 '14 at 02:38

Kevin Brown-Silva

40,873
40
203
237

15

For some reason this leads me to 400 error. The error returned is {"file":["The submitted file is empty."]}. – Divick Feb 04 '16 at 05:52
3

Note that if you don't want to allocate a tempfile for whatever reason (perf would be one), you can also do `tmp_file = BytesIO(b'some text')`; this will give you a binary stream that can be passed as a file object. (https://docs.python.org/3/library/io.html). – Symmetric Apr 21 '16 at 21:41
5

Had to add `tmp_file.seek(0)` before the `post`, but otherwise perfect! This almost drove me nuts so thanks! – jaywink Jul 31 '17 at 23:07
where's the `Image` module coming from? – Feb 10 '19 at 18:51
1

@RudolfOlah `Image` comes from the Pillow library. See https://pillow.readthedocs.io/en/stable/. – Clinton Blackburn Jul 06 '19 at 03:59
It is important to pay attention that it is `format=multipart` and not `content-type`. I will leave this here if someone is also making the same mistake and getting a `415` – nck Jun 10 '21 at 22:00

Meistro · Answer 2 · 2017-01-10T18:12:09.193

22

Python 3 users: make sure you open the file in mode='rb' (read,binary). Otherwise, when Django calls read on the file the utf-8 codec will immediately start choking. The file should be decoded as binary not utf-8, ascii or any other encoding.

# This won't work in Python 3
with open(tmp_file.name) as fp:
        response = self.client.post('my_url', 
                                   {'image': fp}, 
                                   format='multipart')

# Set the mode to binary and read so it can be decoded as binary
with open(tmp_file.name, 'rb') as fp:
        response = self.client.post('my_url', 
                                   {'image': fp}, 
                                   format='multipart')

edited Jan 10 '17 at 18:12

answered May 17 '15 at 19:23

Meistro

3,664
2
28
33

4

I think that `{'image': data}` ought to be `{'image': fp}` in the answer. I was struggling with uploads until I found this post, yet my tests did not pass until I put the file handle object `fp` in place of `data` in the `{'image': data}` dictionary described above. (`{'image': fp}` worked in my case, `{'image': data}` did not.) – dmmfll Aug 17 '15 at 17:43
Updated. Thanks DMfll. – Meistro Jan 10 '17 at 18:12

score 14 · Answer 3 · edited May 04 '22 at 14:44

You can use Django built-in SimpleUploadedFile:

from django.core.files.uploadedfile import SimpleUploadedFile

class TestFileUpload(APITestCase):
    ...

    def test_file_is_accepted(self):
        ...

       tmp_file = SimpleUploadedFile(
                      "file.jpg", "file_content", content_type="image/jpg")

       response = self.client.post(
                      'my_url', {'image': tmp_file}, format='multipart')
       self.assertEqual(response.status_code, status.HTTP_201_CREATED)

score 6 · Answer 4 · answered Jun 01 '18 at 17:19

It's not so simple to understand how to do it if you want to use the PATCH method, but I found the solution in this question.

from django.test.client import BOUNDARY, MULTIPART_CONTENT, encode_multipart

with open(tmp_file.name, 'rb') as fp:
    response = self.client.patch(
        'my_url', 
        encode_multipart(BOUNDARY, {'image': fp}), 
        content_type=MULTIPART_CONTENT
    )

score 2 · Answer 5 · edited May 23 '17 at 12:26

For those in Windows, the answer is a bit different. I had to do the following:

resp = None
with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as tmp_file:
    image = Image.new('RGB', (100, 100), "#ddd")
    image.save(tmp_file, format="JPEG")
    tmp_file.close()

# create status update
with open(tmp_file.name, 'rb') as photo:
    resp = self.client.post('/api/articles/', {'title': 'title',
                                               'content': 'content',
                                               'photo': photo,
                                               }, format='multipart')
os.remove(tmp_file.name)

The difference, as pointed in this answer (https://stackoverflow.com/a/23212515/72350), the file cannot be used after it was closed in Windows. Under Linux, @Meistro's answer should work.

On OSX10.12.5 I got `ValueError: I/O operation on closed file` — joe, Sep 20 '17 at 04:49
I don't think you need the `tmp_file.close()` with the `with` statement. — LondonAppDev, Mar 01 '18 at 17:25

How can I test binary file uploading with django-rest-framework's test client?

5 Answers5

Linked