3

I have a server running flask and some api open for client to upload file.

The client is having problem when uploading big file(>100MB).

At first, I tried urllib2 using the following code at client:

def uploadFile(url, params, user, token):
    try:
        from poster.encode import multipart_encode
        from poster.streaminghttp import register_openers
        import urllib2
        file_path = params["filepath"]
        register_openers()
        tempparams = {'file': open(file_path, "rb")}
        f = open(file_path,'rb')
        tempparams = {'file': open(file_path, "rb")}
        finalparams = dict(params, **tempparams)
        datagen, headers = multipart_encode(finalparams)
        request = urllib2.Request(url, datagen, headers)
        request.add_header("Authorization", "Basic %s" % base64string)
        result = urllib2.urlopen(request)
        resultData =  result.read()
        resultJson = json.loads(resultData)
        return resultJson
    except Exception as e:
        return {"error": e, "error_code": 503}

At server:

def upload_file_old():
    fileobj =request.files["file"]

    try:
        fileobj.save(webserver_filepath)
    except Exception as e:
        logger.info(e)
    .....
    return some_result, 201

But the above code will lead to the following error when uploading big file at client with server recording no log at all. 

urlopen error [errno 10054]

Then I tried using requests toolbelt to send file since many post at stack overflow suggest using request other than using urllib.

At client:

def uploadFile(url, params, user, token):
    try:
        import requests
        from requests_toolbelt import MultipartEncoder
        file_path = params["filepath"]
        m = MultipartEncoder(fields={
        'file':(params["name"],open(file_path,'rb'),"application/octet-stream" )
        }) 
        r = requests.post(url, data=m, headers={'Content-Type': m.content_type}, auth=(user, token))
        return r.json()
    except Exception as e:
        return {"error": e, "error_code": 503}

At server:

def upload_file_old():
    fileobj =request.files["file"]

    try:
        fileobj.save(webserver_filepath)
    except Exception as e:
        logger.info(e)
    .....
    return some_result, 201

Again, sending small file works fine while big file failed for the above code, the error output is as foloowing:

 ('Connection aborted.', error(104, 'Connection reset by peer'))

What would be the correct way to upload big file in a restful http request using python?

Heyang Wang
  • 360
  • 2
  • 4
  • 19
  • 2
    do you connect to the running python app directly or are you behind some other www server (e.g. behind nginx)? – grepe Oct 24 '16 at 14:11
  • The serve is behind ngix, oh, as I am typing, connecting to server without nigx and send big file works fine without error. Dude, thanks so much! Do you know why would ngix causing this problem? – Heyang Wang Oct 24 '16 at 14:28
  • 2
    100MB is the default value of nginx config parameter (probably some other servers too) of maximum file size for upload. take look here at how to increase it: https://easyengine.io/tutorials/php/increase-file-upload-size-limit/ (it's for php, but the nginx part should be the same). – grepe Oct 24 '16 at 14:46
  • Thanks ! I guess next time I shouldn't only focusing on python code itself when debugging. – Heyang Wang Oct 24 '16 at 14:54
  • Possible duplicate of [Using Python Requests to send file and JSON in single request](http://stackoverflow.com/questions/24642040/using-python-requests-to-send-file-and-json-in-single-request) – James Shewey May 05 '17 at 21:26
  • Actually, the problem is caued by nginx and had no similarity with the link you posted. – Heyang Wang May 07 '17 at 07:50

0 Answers0