Django rest framework large file upload

Question

Is there an easy way to upload large files from the client side to a django rest framework endpoint. In my application, users will be uploading very large files (>4gb). Browsers have a upload limit, here's the chart.

My current idea is to upload the file in chunks from the client side and receive the chunks from the rest endpoint. But how will I do that? I saw some libraries like - resumable.js, tus.js, flow.js etc. But how will I handle the chunks in the backend? Is there any library that is actively maintained for a problem like this? Please help me.

I recommend using a sftp-server. I am assuming your app works synchronously. If you request django with that amount of data, you will block all other incomming requests until the file is uploaded even as chunks. — Klim Bim, Jul 23 '21 at 07:09
@KlimBim My app is asynchronous. React as the frontend and django as the backend. sftp-server is not a great option for this situation. I saw some implementations of tusjs server in django, however- the repo for that was very old. It has a lot of problems as it was implemented 6 years ago. The documentation of tus protocol is also not very good. — Pranava Mohan, Jul 23 '21 at 07:35

score 2 · Answer 1 · answered Jul 26 '21 at 21:53

Maybe this module could help: https://github.com/jkeifer/drf-chunked-upload. The module is utilized into a sample django app at the link, with example code for implementation. Here is the typical usage case the module provides, without the sample code for simplicity (code is at the link if you want it):

An initial PUT request is sent to the url linked to ChunkedUploadView (or any subclass) with the first chunk of the file. The name of the chunk file can be overriden in the view (class attribute field_name).
In return, the server will respond with the url of the upload, and the current offset.

3 Repeatedly PUT subsequent chunks to the url returned from the server.

Server will continue responding with the url and current offset.
Finally, when upload is completed, POST a request to the returned url. This request must include the checksum (hex) of the entire file.
If everything is OK, server will response with status code 200 and the data returned in the method get_response_data (if any).
If you want to upload a file as a single chunk, this is also possible! Simply make the first request a POST and include the checksum digest for the file. You don't need to include the Content-Range header if uploading a whole file.

Based on these instructions, it seems that the server handles the upload by tracking offsets of the chunk through received headers ("Content-Range"), as well as its url, storing the uploaded chunks in .part files. It then responds like so:

   {
        'id': 'f64ebd67-83a3-45b6-8acd-c749ea1ed4cd'
        'url': 'https://your-host/<path_to_view>/f64ebd67-83a3-45b6-8acd-c749ea1ed4cd',
        'file': 'https://your-host/<path_to_file>/f64ebd67-83a3-45b6-8acd-c749ea1ed4cd.part',
        'filename': 'example.bin',
        'offset': 10000,
        `created_at`: '2021-05-18T17:12:50.318718Z',
        'status': 1,
        'completed_at': None,
        'user': 1
    }

When the full file is uploaded as determined by the recieved headers, the .part files are combined into the final upload. This also allows you to resume uploads if they are interuptted, because the existing .part files persist until the upload finishes.

Is there any JS client built for this? Or should we implement our own? — Pranava Mohan, Jul 27 '21 at 01:02
I am using react. If there is already a JS client built for this, it would be awesome. It would make my job a lot easier. — Pranava Mohan, Jul 28 '21 at 00:16
This specific package doesn't have one built for it, but there are plenty of online tutorials for writing your own, e.g. (https://www.section.io/engineering-education/react-and-django-rest-framework/, https://www.valentinog.com/blog/drf/) that you can look at to easily integrate drf with react. Combine that with the answer above and you should be good to go. — figbar, Jul 28 '21 at 15:11

score 0 · Answer 2 · answered Jul 23 '21 at 00:01

0

https://stackoverflow.com/a/26278960/12776116

Maybe this can help you. As you mentioned, the file is uploaded by breaking it into small parts.

answered Jul 23 '21 at 00:01

xxxecute

194
2
9

But that doesn't answer on how I can handle the chunks in the backend. Also the answer had broken links. – Pranava Mohan Jul 23 '21 at 00:22
https://github.com/juliomalegria/django-chunked-upload – xxxecute Jul 23 '21 at 11:44

score 0 · Answer 3 · answered Jul 27 '21 at 12:41

0

you should do it with celery tasks.

take a look at this link. it explains how to upload a file using django and celery.

answered Jul 27 '21 at 12:41

Mojtaba Arezoomand

2,140
8
23

Django rest framework large file upload

3 Answers3