1

I'm in the process of writing a python module to POST files to a server , I can upload files of size of upto 500MB but when I tried to upload a 1gb file the upload failed, If I were to use something like cURL it won't fail. I got the code after googling how to upload multipart formdata using python , the code can be found here. I just compiled and ran that code , the error I'm getting is this 

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    opener.open("http://127.0.0.1/test_server/upload",params)
  File "C:\Python27\lib\urllib2.py", line 392, in open
    req = meth(req)
  File "C:\Python27\MultipartPostHandler.py", line 35, in http_request
    boundary, data = self.multipart_encode(v_vars, v_files)
  File "C:\Python27\MultipartPostHandler.py", line 63, in multipart_encode
    buffer += '\r\n' + fd.read() + '\r\n'  
MemoryError

I'm new to python and having a hard time grasping it. I also came across another program here , I'll be honest I don't know how to run it. I tried running it by guessing based on the function name , but that didn't work.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
cyberbemon
  • 3,000
  • 11
  • 37
  • 62
  • Maybe I'm misunderstanding something, but a 1 gb upload is supposed to fail if the server's limit is 500 megabytes. – kevin628 May 24 '12 at 15:28
  • I didn't set any limit , like I said I can upload 1gb + files using cURL. – cyberbemon May 24 '12 at 15:39
  • @cyberbemon: the traceback shows that the code tries to load the whole file in memory (`fd.read()`). Either increase an allowed amount of memory for the process or stream the content (as `curl` probably does) see http://stackoverflow.com/questions/2502596/python-http-post-a-large-file-with-streaming – jfs May 24 '12 at 15:51

1 Answers1

5

The script in question isn't very smart and builds the POST body in memory.

Thus, to POST a 1GB file, you'll need 1GB of memory just to hold that data, plus the HTTP headers, boundaries, and python and the code itself.

You'd have to rework the script to use mmap instead, where you first construct the whole body in a temp file before handing that file wrapped in a mmap.mmap value to passing it to request.add_data.

See Python: HTTP Post a large file with streaming for hints on how to achieve that.

Community
  • 1
  • 1
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thanks I'll have a look at it and post the updates , Do you by any chance know how to run the program posted in the second link ? – cyberbemon May 25 '12 at 13:23
  • There was 2 links posted in the question [here](http://code.activestate.com/recipes/146306/) is the one I'm talking about – cyberbemon May 25 '12 at 14:55
  • The recipe you link to doesn't stream files either, but requires reading into memory too. Not sure if that's the link you meant. – Martijn Pieters May 25 '12 at 14:59
  • ah well its no use then , back to dissecting the other code. Python is confusing ! – cyberbemon May 25 '12 at 15:07