How to upload large (~100Mb) files?

Question

I have following flask app

fileform.html:

<html>
    <head>
        <title>Simple file upload using Python Flask</title>
    </head>
    <body>
        <form action="/getSignature" method="post" enctype="multipart/form-data">
          Choose the file: <input type="file" name="photo"/><BR>
              <input type="submit" value="Upload"/>
        </form>
    </body>
</html>

app.py:

import os
from flask import Flask, request, render_template, url_for, redirect


app = Flask(__name__)


@app.route("/")
def fileFrontPage():
    return render_template('fileform.html')

@app.route("/getSignature", methods=['POST'])
def handleFileUpload():
    if 'photo' in request.files:
        photo = request.files['photo']
        if photo.filename != '':
            filepath = os.path.join('/flask/files', photo.filename)
            photo.save(filepath)
    return render_template('result.html')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

This works for smaller files, but fails for larger file. After clicking Upload button, browser shows (Uploading 13%..) and then browser timeout with ERR_CONNECTION_RESET. Don't see any error in flask app.

I am serving this through Nginx. When I check nginx logs I see

2020/02/20 22:38:47 [error] 6#6: *170614 client intended to send too large body: 80762097 bytes, client: 10.2.16.178, server: localhost, request: "POST /getSignature

Is there any Nginx config I need to add for this ?

I want to upload files upto size 100Mb.

How are you running app.py? In the console/an ide or is it being run through a web server like apache/nginx? — Rashid 'Lee' Ibrahim, Feb 20 '20 at 22:39
related: https://stackoverflow.com/questions/44727052/handling-large-file-uploads-with-flask — njzk2, Feb 20 '20 at 22:40

Evhz · Accepted Answer · 2020-02-20T22:56:50.060

1

Probably the content length is limited to a default value.

As you are using nginx, probably the upstream timeout it playing its game (default for upstream keepalive_timeout 60s) add the following setting to nginx:

keepalive_timeout 900s;

#extra
proxy_read_timeout 900s;
uwsgi_read_timeout 900s;

More tunning here

And if you use a python wsgi server like uwsgi, set also the keep alive there:

http-keepalive 900;

edited Feb 20 '20 at 22:56

answered Feb 20 '20 at 22:41

Evhz

8,852
9
51
69

Adding `client_max_body_size 100M;` to Nginx config fixed the issue – roy Feb 21 '20 at 14:40

score 1 · Answer 2 · answered Feb 20 '20 at 22:41

1

From https://flask.palletsprojects.com/en/1.1.x/patterns/fileuploads/ :

Connection Reset Issue

When using the local development server, you may get a connection reset error instead of a 413 response. You will get the correct status response when running the app with a production WSGI server.

Solution: don't use the development server, setup a real WSGI server

answered Feb 20 '20 at 22:41

njzk2

38,969
7
69
107

Link to the deployment options and I'll happily upvote – roganjosh Feb 20 '20 at 22:42
1

https://flask.palletsprojects.com/en/1.1.x/deploying/ but I feel there is something else in this problem – roganjosh Feb 20 '20 at 22:43
@roganjosh turns out it's not related, as the OP is running behind nginx and that's where the connection reset happens – njzk2 Feb 21 '20 at 17:54

score 1 · Answer 3 · answered Feb 20 '20 at 22:47

Have you looked into chunking? It allows you to break up files and data into discrete pieces to send over the wire. And you'll have two main parts: the front-end and the backend. For the front-end you can use something like Dropzone.js. However you'll need to enable the chunking behavior, as by default it isn't included. Luckily, it is really easy to enable.

This might be completed like so:

<html>
    <meta charset="UTF-8">
    <link rel="stylesheet"
     href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css"/>

    <link rel="stylesheet"
     href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/basic.min.css"/>

    <script type="application/javascript"
     src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js">
    </script>
    <head>
        <title>Simple file upload using Python Flask</title>
    </head>
        <form method="POST" action='/upload' class="dropzone dz-clickable"
                    id="dropper" enctype="multipart/form-data">
        </form>

        <script type="application/javascript">
                Dropzone.options.dropper = {
                        paramName: 'file',
                        chunking: true,
                        forceChunking: true,
                        url: '/upload',
                        maxFilesize: 1025, // megabytes
                        chunkSize: 1000000 // bytes
                }
        </script>
</html>

And the following flask example will handle the back-end:

import logging
import os

from flask import render_template, Blueprint, request, make_response
from werkzeug.utils import secure_filename

from pydrop.config import config

blueprint = Blueprint('templated', __name__, template_folder='templates')

log = logging.getLogger('pydrop')


@blueprint.route('/')
@blueprint.route('/index')
def index():
    # Route to serve the upload form
    return render_template('index.html',
                           page_name='Main',
                           project_name="pydrop")


@blueprint.route('/upload', methods=['POST'])
def upload():
    file = request.files['file']

    save_path = os.path.join(config.data_dir, secure_filename(file.filename))
    current_chunk = int(request.form['dzchunkindex'])

    # If the file already exists it's ok if we are appending to it,
    # but not if it's new file that would overwrite the existing one
    if os.path.exists(save_path) and current_chunk == 0:
        # 400 and 500s will tell dropzone that an error occurred and show an error
        return make_response(('File already exists', 400))

    try:
        with open(save_path, 'ab') as f:
            f.seek(int(request.form['dzchunkbyteoffset']))
            f.write(file.stream.read())
    except OSError:
        # log.exception will include the traceback so we can see what's wrong 
        log.exception('Could not write to file')
        return make_response(("Not sure why,"
                              " but we couldn't write the file to disk", 500))

    total_chunks = int(request.form['dztotalchunkcount'])

    if current_chunk + 1 == total_chunks:
        # This was the last chunk, the file should be complete and the size we expect
        if os.path.getsize(save_path) != int(request.form['dztotalfilesize']):
            log.error(f"File {file.filename} was completed, "
                      f"but has a size mismatch."
                      f"Was {os.path.getsize(save_path)} but we"
                      f" expected {request.form['dztotalfilesize']} ")
            return make_response(('Size mismatch', 500))
        else:
            log.info(f'File {file.filename} has been uploaded successfully')
    else:
        log.debug(f'Chunk {current_chunk + 1} of {total_chunks} '
                  f'for file {file.filename} complete')

    return make_response(("Chunk upload successful", 200))

+1 for suggesting a much more reasonable solution than the "raise the cap and increase the timeouts" that gets suggested everywhere — mbrig, Mar 06 '20 at 20:31

How to upload large (~100Mb) files?

3 Answers3