Flask File upload - Getting only partial file

Question

I am trying to upload a CSV file of 60 mb to my application. But when i save the file to server , it only contains partial(25 mb) of total records from the original. This issue is occurring only randomly.

@fileupload.route('/loadfile',methods = ['POST'])
def store_to_db():
for _file in request.files.getlist('filelist'):
        name, ext = FileUtil.get_filename_without_ext(_file.filename)#fetching file name and extension
        fd, path = tempfile.mkstemp(suffix=ext, prefix=name)#saving file as temp
        os.write(fd, _file.read())
        try:
            file_upload_service.savefiletodb(path)
        except Exception as e:
            logger.info(str(e))
        os.close(fd)

Uploader side code:

    data= {}
    url_path = 'localhost:5000/loadfile'
    data['filelist']=open('employe.txt', 'rb')
    resp = requests.post(url=url_path, files=data)

Any idea why its behaving like this ?

Note:

When i upload through postman, i am not facing any issue.
If I upload through code , sometimes I get partial file.
file object in log , when upload via postman

'employee.TXT' ('text/plain')>
file object in log , when upload via code

'employee.TXT' ('None')>

BWT: in Python you have to use `#` instead of `//` to create comment in code. Using `//` you try to divide and it can be missleading for people which reads your code in answer. — furas, Jun 17 '20 at 15:46
don't you get any error message in logger or when you run it in console ? Do you save it in database ? Did you create field in database which can keep 60MB ? OR maybe you get error in DevTool in Firefox/Chrome when you upload file? — furas, Jun 17 '20 at 15:50
did you check in Google `"flask upload big file"` ? I using this in Google found `app.config['MAX_CONTENT_LENGTH']` in [Improving Uploads](https://flask.palletsprojects.com/en/1.1.x/patterns/fileuploads/#improving-uploads) — furas, Jun 17 '20 at 15:52
@furas Yes checked, but by default flask accepts file of unlimited size. — Joby Wilson Mathews, Jul 07 '20 at 06:00
You can use https://github.com/flowjs/flow.js and make the browser chunk the files and flask will have all the chunks at once — bigbounty, Jul 07 '20 at 06:11
are using dedicated server (nginx..etc.) or development server? — adnanmuttaleb, Jul 07 '20 at 06:24
both upload (postman, through-code) is done to same environment ? — adnanmuttaleb, Jul 07 '20 at 06:26
maybe you need to check the network and see if files are sending in right size/chunk or not and see also if function/saving part is able to handle data when there is network delay — sahasrara62, Jul 07 '20 at 06:30
try to set the `content-type` header, for example: `headers={'content-type': 'text/plain'}` — adnanmuttaleb, Jul 07 '20 at 06:35
Your provided code does not work for me. The client code does not appear to match the `requests.post` interface. `files` should be a dictionary, and the url doesn't have `http`. Here is my attempt at replicating your setup to test https://gist.github.com/Multihuntr/22455c371451cd4063763c0b26230bd2. This gist works fine for me 100% of the time. — Multihunter, Jul 07 '20 at 07:38
Can you update the question to indicate if it is just the temporary file that is the wrong size, or if it is just the file in the database that is the wrong size, or if it is both? (I'm assuming both, but, it would be good to be clear) — Multihunter, Jul 07 '20 at 08:14
I've made my gist as similar to your code as possible and it still works. The problem is not within the provided code. The problem seems to be somewhere in your setup, so you'll have to provide details on how the server is set up. The goal is, of course, a MWE. Else it can't be debugged without guessing somewhat randomly. — Multihunter, Jul 07 '20 at 08:36

score 3 · Answer 1 · answered Jul 09 '20 at 06:33

If the issue is with the file size which is exceeding the file size limit, Flask will raise a RequestEntityTooLarge exception.

If this is the case, you can simply configure the MAX_CONTENT_LENGTH config key. For example,

from flask import Flask, Request

app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 20 * 1024 * 1024

The above code will set the File upload limit to 20 megabytes. If the user tries to upload a file larger than the set limit (in this case 20 MB), Then Flask will raise a RequestEntityTooLarge exception.

And then if required, Allowed file extensions can also be specified by,

ALLOWED_EXTENSIONS = {'txt', 'docx','pdf', 'png', 'jpg', 'jpeg', 'gif'}

For more help, Refer the Flask Docs for Uploading files.

score 1 · Answer 2 · answered Jul 09 '20 at 06:04

1

try to set app.config['MAX_CONTENT_LENGTH'] here
try to set headers like return Response(mimetype='text/plain') here is reference for the same
try this tutorial

answered Jul 09 '20 at 06:04

M_x

782
1
8
26

Flask File upload - Getting only partial file

2 Answers2