8

I'm playing around with the Python Requests module that has so far been a delight.

However, I've run into an issue whilst attempting to post a zip file using multipart/form-data.

I'm using Digest authentication and have been able to successfully post other file types e.g. .xls etc.

I'm creating a post request using:

file = open('/Users/.../test.zip', 'rb').read()
r = requests.post(url, auth=HTTPDigestAuth('dev', 'dev'), data = {"mysubmit":"Go"}, files={"archive": ("test.zip", file)})

This errors out and gives:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.2.2.70', port=80): Max retries exceeded with url: /plugin_install 
(Caused by <class 'socket.error'>: [Errno 32] Broken pipe)

I've tried with smaller size zip files and changing the data/files values, and the same error occurs.

Am I missing something obvious?

Thanks for any light you can shed!

Benji Barash
  • 423
  • 3
  • 7
  • 15

2 Answers2

15

As far as requests is concerned, there is no difference between a zip file and any other binary blob of data.

Your server is broken here; it is cutting of the connection when you send it a zip file. That is not something requests can do anything about.

You may want to test against http://httpbin.org/ when you run into problems like these; it is a testing service built by the author of the requests library.

Another tip: you don't need to read the whole file object into memory when sending. Just pass the object itself to requests instead:

fileobj = open('/Users/.../test.zip', 'rb')
r = requests.post(url, auth=HTTPDigestAuth('dev', 'dev'), data = {"mysubmit":"Go"}, files={"archive": ("test.zip", fileobj)})

Demo against httpbin.org:

>>> import requests
>>> fileobj = open('/tmp/test.zip', 'rb')
>>> r = requests.post('http://httpbin.org/post', data={"mysubmit":"Go"}, files={"archive": ("test.zip", fileobj)})
>>> r
<Response [200]>
>>> r.json()
{u'origin': u'217.32.203.188', u'files': {u'archive': u'data:application/zip;base64,<long base64 body omitted>'}, u'form': {u'mysubmit': u'Go'}, u'url': u'http://httpbin.org/post', u'args': {}, u'headers': {u'Content-Length': u'57008', u'Accept-Encoding': u'gzip, deflate, compress', u'Connection': u'close', u'Accept': u'*/*', u'User-Agent': u'python-requests/1.2.3 CPython/2.7.5 Darwin/12.4.0', u'Host': u'httpbin.org', u'Content-Type': u'multipart/form-data; boundary=9aec1d03a1794177a38b48416dd4c811'}, u'json': None, u'data': u''}
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Note that you can send the file object directly and still use a tuple to provide the filename. – Lukasa Aug 13 '13 at 11:54
  • 1
    @Lukasa: a quick re-test and it's clear I made a mistake; I must've already read the file object and forgot to seek back to 0.. – Martijn Pieters Aug 13 '13 at 12:17
  • Thanks for the response! Your clarification has definitely helped as I can now post small zip files successfully. I still seem to be having that error when posting any zip larger than around 98 KB though. Is this potentially a requests/urllib3 timeout/content length issue? – Benji Barash Aug 13 '13 at 13:31
  • @BenjiBarash: no, that is still a server-side issue. – Martijn Pieters Aug 13 '13 at 13:34
  • I was previously using a library called Poster that didn't prompt these issues with the server though. Poster called a function multipart_encode and then I'd use urllib2.Request with the data it generated. I found Requests instead, because Poster had issues with multipart/form-data using Digest Auth. https://bitbucket.org/chrisatlee/poster/issue/7/multipart-form-post-doesnt-work-with – Benji Barash Aug 13 '13 at 13:40
  • Just found this on an older question you were active on - it seems to have fixed the issue...I think. http://stackoverflow.com/a/16604431/141352 – Benji Barash Aug 13 '13 at 13:58
  • @BenjiBarash: You mean you were opening the files in text mode, not binary mode? – Martijn Pieters Aug 13 '13 at 14:02
  • @BenjiBarash: Ah, no, you were not using a session and thus were retrying the send after authentication succeeded. – Martijn Pieters Aug 13 '13 at 14:03
  • Yeah, this seems to have worked: `file = open('test.zip', 'rb') sess = requests.Session() sess.auth = HTTPDigestAuth('dev', 'dev') sess.get(url) sess.post(url, data = {"mysubmit":"Go"}, files={"archive": ("test.zip", file)})` Firing that initial get(url) within the session looks to have been key to the Digest protocol. – Benji Barash Aug 13 '13 at 14:08
1

If you are facing any errors while uploading the zip: This problem could be caused by 'Content-Type': 'multipart/form-data' setting on the header. The problem can be solved by deleting this setting, as in this example:

header = {'Content-Type': 'multipart/form-data', 'Authorization': 'Bearer {}'.format(bearerToken)}

change it to:

header = {'Authorization': 'Bearer {}'.format(bearerToken)}

Code which worked for me:

 header = {"INFA-SESSION-ID":self._v3SessionID}
 files = {'package': ("response.zip", open("C:\Users\myUser\Downloads\response.zip", 'rb'),'application/zip')}
 response = re.post(url, headers=header, files=files)
Prashanth Pradeep
  • 173
  • 2
  • 6