11

I cannot for the life of me figure out how to perform an HTTP PUT request with verbatim binary data in Python 2.7 with the standard Python libraries.

I thought I could do it with urllib2, but that fails because urllib2.Request expects its data in application/x-www-form-urlencoded format. I do not want to encode the binary data, I just want to transmit it verbatim, after the headers that include 

Content-Type: application/octet-stream
Content-Length: (whatever my binary data length is)

This seems so simple, but I keep going round in circles and can't seem to figure out how.

How can I do this? (aside from open up a raw binary socket and write to it)

Community
  • 1
  • 1
Jason S
  • 184,598
  • 164
  • 608
  • 970

4 Answers4

12

I found out my problem. It seems there is some obscure behavior in urllib2.Request / urllib2.urlopen() (at least in Python 2.7)

The urllib2.Request(url, data, headers) constructor seems to expect the same type of string in its url and data parameters.

I was giving the data parameter raw data from a file read() call (which in Python 2.7 returns it in the form of a 'plain' string), but my url was accidentally Unicode because I concatenated a portion of the URL from the result of another function which returned Unicode strings.

Rather than trying to "downcast" url from Unicode -> plain strings, it tried to "upcast" the data parameter to Unicode, and it gave me a codec error. (oddly enough, this happens on the urllib2.urlopen() function call, not the urllib2.Request constructor)

When I changed my function call to

# headers contains `{'Content-Type': 'application/octet-stream'}`
r = urllib2.Request(url.encode('utf-8'), data, headers)

it worked fine.

Uli Köhler
  • 13,012
  • 16
  • 70
  • 120
Jason S
  • 184,598
  • 164
  • 608
  • 970
  • I bumped into the [requests library](https://github.com/kennethreitz/requests) just today. In the future, you might want to use that. – Spencer Rathbun Jan 03 '12 at 18:22
9

You're misreading the documentation: urllib2.Request expects the data already encoded, and for POST that usually means the application/x-www-form-urlencoded format. You are free to associate any other, binary data, like this:

import urllib2

data = b'binary-data'
r = urllib2.Request('http://example.net/put', data,
                    {'Content-Type': 'application/octet-stream'})
r.get_method = lambda: 'PUT'
urllib2.urlopen(r)

This will produce the request you want:

PUT /put HTTP/1.1
Accept-Encoding: identity
Content-Length: 11
Host: example.net
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

binary-data
phihag
  • 278,196
  • 72
  • 453
  • 469
  • 1
    But what I get is `UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)` – Jason S Jan 02 '12 at 23:00
  • btw, how did you get the raw request sent from urllib2? – Jason S Jan 02 '12 at 23:02
  • @JasonS That's because your data is a *string* (a `unicode` object), not a `bytes` object. Encode it to the correct encoding with [`encode`](http://docs.python.org/library/stdtypes.html#str.encode). It can be helpful to write the program in Python 3 and then backport it to 2.x. – phihag Jan 02 '12 at 23:07
  • @JasonS Sorry, I don't understand your question. The code I used is right there in the answer. Notice the `b` in front of `b'binary-data'`, which makes it a `bytes` literal (although that has no effect in 2.x). Can you post the code that throws the `UnicodeDecodeError`? – phihag Jan 02 '12 at 23:09
  • I'll try to post a test case. For the record, I am *not* misreading the documentation. I understand about urllib2.Request; that was made clear in my other question. My question is how to use *any* Python libraries to `PUT` unencoded binary data. – Jason S Jan 03 '12 at 14:30
  • @JasonS Well, of course it depends on the library you're using, but with urllib(2), you just set the `data` parameter of Request to the `bytes` object (= the binary data). Your problem is that you have a *string* (`unicode`), and need to convert it `bytes` (confusingly identical to `str` in Python 2.x, which despite the name is *not* a string). – phihag Jan 03 '12 at 14:35
4

Have you considered/tried using httplib?

HTTPConnection.request(method, url[, body[, headers]])

This will send a request to the server using the HTTP request method method and the selector url. If the body argument is present, it should be a string of data to send after the headers are finished. Alternatively, it may be an open file object, in which case the contents of the file is sent; this file object should support fileno() and read() methods. The header Content-Length is automatically set to the correct value. The headers argument should be a mapping of extra HTTP headers to send with the request.

Adam Wagner
  • 15,469
  • 7
  • 52
  • 66
1

This snipped worked for me to PUT an image:

on HTTPS site. If you don't need HTTPS, use httplib.HTTPConnection(URL) instead.

import httplib
import ssl
API_URL="api-mysight.com"
TOKEN="myDummyToken"
IMAGE_FILE="myimage.jpg"
imageID="myImageID"
URL_PATH_2_USE="/My/image/" + imageID +"?objectId=AAA"
headers = {"Content-Type":"application/octet-stream", "X-Access-Token": TOKEN}
imgData = open(IMAGE_FILE, "rb")
REQUEST="PUT"
conn = httplib.HTTPSConnection(API_URL, context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))
conn.request(REQUEST, URL_PATH_2_USE, imgData, headers)
response = conn.getresponse()
result = response.read()
serg_sh
  • 11
  • 1