Django - Larger files failing to upload to heroku

Question

I am having trouble doing a post to a django app hosted on heroku. I have a form that submits 3 images and everything works fine when I use smaller images of about 100kb, however when I user larger images of ~3MB the upload fails with error in heroku logs showing as at=error code=H13 desc="Connection closed without response" method=POST path="/"

In django, am simply saving the images then emailing them as shown below, where formdata is holding the images. Hope the snippet is enough:

for each in form_data:
    pic = form_data[each]
    if pic:
        filename = os.path.join(self.location,f"{i} - {pic.name}")
        imgbytes = pic.read()
        with open(filename, 'wb+') as destination:
            destination.write(imgbytes)

        i+=1
        fileholder.append(filename)
email = EmailMessage(
        subject = 'Hello',
        body = 'Body goes here',
        from_email = 'example@yahoo.com',
        to = ['test@google.com'],
        )

for each in fileholder:
    email.attach_file(each)
email.attach_file(newpath)
email.send()

What is causing this and how can i ensure any image size is uploaded succesfully?

Hi, what version of django are you using ? – MSR974 Aug 05 '20 at 08:32 — MSR974, Aug 05 '20 at 08:32

score 1 · Answer 1 · answered Aug 17 '20 at 07:50

Ok so I found out all requests on heroku will timeout after 30sec which is what was happening in my case. The uploads where taking longer than that and my solution was to process the uploads client side and upload directly to S3 instead.

More info here

MSR974 · Answer 2 · 2020-08-05T08:41:25.203

0

Django has a settings DATA_UPLOAD_MAX_MEMORY_SIZE which prevent file larger than 2.5MB (Default: 2621440 (i.e. 2.5 MB)) to be uploaded and raises a SuspiciousOperation (RequestDataTooBig) Exception

Try changing it in your settings.py

edited Aug 05 '20 at 08:41

answered Aug 05 '20 at 08:35

MSR974

632
3
12

Hi I increased to `DATA_UPLOAD_MAX_MEMORY_SIZE = 262144000` but error still persisted. – West Aug 06 '20 at 01:13
I also set `FILE_UPLOAD_MAX_MEMORY_SIZE = 262144000` and still the same error. Everything works fine when testing locally though so not sure why. – West Aug 06 '20 at 02:00
1

is your server behind an nginx webserver ? it could be nginx which is limiting your [uploads](https://stackoverflow.com/questions/26717013/how-to-edit-nginx-conf-to-increase-file-size-upload) – MSR974 Aug 06 '20 at 08:26
Im just using the free heroku tier and not sure if there's nginx or how to edit it. I tried digital ocean too and this also happens even before i set up any nginx. – West Aug 10 '20 at 07:00

score 0 · Answer 3 · answered Aug 05 '20 at 09:13

0

As my know,almost all web-frameworks will limit the data's size in one request. such as Django,Tornado,Spring.

And it is not safe,if not limit the max size of one request.

As the MSR974 say,change the DATA_UPLOAD_MAX_MEMORY_SIZE,it is a way to change the setting about the max size of one request.But in your question:ensure any image size is uploaded succesfully.Key words is any size

Have a try on multipart upload ?

Which is by the way of spliting one file into two or more files and merging them with order after upload to backend.(hope I explaned it clearly)

answered Aug 05 '20 at 09:13

lanhao945

427
5
21

Hi will try look into this, I assume the splits have to be done with javascript but I'm not great at it. Will research. The app is to be used to register people and they upload a few images for identification. Each of the images uploaded are usually small about 300kb but every now and then someone has a 2-3MB file. – West Aug 06 '20 at 01:23
@West Of course , splits should use javascript. If you are unfamiliar at javascript,try to do like the MSR974 say. Also, another way is try to compress a picture or modify size before upload(: just know how to use js,but not very good at js can do.）. about three ways to solve the issue.if you hope not use js.make sure the pictures which people will upload could not larger than you set.key words `any size` can not provided. – lanhao945 Aug 06 '20 at 02:46
Thanks will give that a go., ddnt know uploads get this complex so fast. – West Aug 06 '20 at 04:19

Django - Larger files failing to upload to heroku

3 Answers3