Does Python-Requests support OrderedDicts, or is something else going wrong here?

Question

I'm trying to POST a request to an Amazon S3 endpoint using Python's Requests library. The request is of the multipart/form-data variety, because it includes the POSTing of an actual file.

One requirement specified by the API I'm working against is that the file parameter must be posted last. Since Requests uses dictionaries to POST multipart/form-data, and since dictionaries don't follow a dictated order, I've converted it into an OrderedDict called payload. It looks something like this before POSTing it:

{'content-type': 'text/plain',
 'success_action_redirect':     'https://ian.test.instructure.com/api/v1/files/30652543/create_success?uuid=<opaque_string>',
 'Signature': '<opaque_string>',
 'Filename': '',
 'acl': 'private',
 'Policy': '<opaque_string>',
 'key': 'account_95298/attachments/30652543/log.txt',
 'AWSAccessKeyId': '<opaque_string>',
 'file': '@log.txt'}

And this is how I POST it:

r = requests.post("https://instructure-uploads.s3.amazonaws.com/", files = payload)

The response is a 500 error, so I'm really not sure what the issue is here. I'm just guessing that it has to do with my use of OrderedDict in Requests—I couldn't find any documentation suggesting Requests does or doesn't support OrderedDicts. It could be something completely different.

Does anything else stick out to you that would cause the request to fail? I could provide more detail if need be.

Okay, update, based on Martijn Pieters' earlier comments:

I changed the way I'm referencing the log.txt file by adding it to the already created upload_data dictionary like this:

upload_data['file'] = open("log.txt")

pprinting the resulting dictionary I get this:

{'AWSAccessKeyId': '<opaque_string>',
 'key': '<opaque_string>',
 'Policy': '<opaque_string>',
 'content-type': 'text/plain',
 'success_action_redirect': 'https://ian.test.instructure.com/api/v1/files/30652688/create_success?uuid=<opaque_string>',
 'Signature': '<opaque_string>',
 'acl': 'private',
 'Filename': '',
 'file': <_io.TextIOWrapper name='log.txt' mode='r' encoding='UTF-8'>}

Does that value for the file key look correct?

When I post it to a RequestBin I get this, which looks pretty similar to Martin's example:

POST /1j92n011 HTTP/1.1
User-Agent: python-requests/1.1.0 CPython/3.3.0 Darwin/12.2.0
Host: requestb.in
Content-Type: multipart/form-data; boundary=e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Length: 2182
Connection: close
Accept-Encoding: identity, gzip, deflate, compress
Accept: */*

--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="AWSAccessKeyId"; filename="AWSAccessKeyId"
Content-Type: application/octet-stream

<opaque_string>
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="key"; filename="key"
Content-Type: application/octet-stream

<opaque_string>
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="Policy"; filename="Policy"
Content-Type: application/octet-stream

<opaque_string>
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="content-type"; filename="content-type"
Content-Type: application/octet-stream

text/plain
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="success_action_redirect"; filename="success_action_redirect"
Content-Type: application/octet-stream

https://ian.test.instructure.com/api/v1/files/30652688/create_success?uuid=<opaque_string>
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="Signature"; filename="Signature"
Content-Type: application/octet-stream

<opaque_string>
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="acl"; filename="acl"
Content-Type: application/octet-stream

private
--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="Filename"; filename="Filename"
Content-Type: application/octet-stream


--e8c3c3c5bb9440d1ba0a5fe11956e28d
Content-Disposition: form-data; name="file"; filename="log.txt"
Content-Type: text/plain

This is my awesome test file.
--e8c3c3c5bb9440d1ba0a5fe11956e28d--

However, I still get a 500 returned when I try to POST it to https://instructure-uploads.s3.amazonaws.com/. I've tried just adding the open file object to files and then submitting all the other values in a separate dict through data, but that didn't work either.

Answer 1

You need to split what you're sending into an OrderedDict passed to data and one sent to files. Right now AWS is (correctly) interpretting your data parameters as FILES, not as form paramaters. It should look like this:

data = OrderedDict([
    ('AWSAccessKeyId', '<opaque_string>'),
    ('key', '<opaque_string>'),
    ('Policy', '<opaque_string>'),
    ('content-type', 'text/plain'),
    ('success_action_redirect', 'https://ian.test.instructure.com/api/v1/files/30652688/create_success?uuid=<opaque_string>'),
    ('Signature', '<opaque_string>'),
    ('acl', 'private'),
    ('Filename', ''),
])

files = OrderedDict([('file', open('log.txt'))])

requests.post(url, data=data, files=files)

Answer 2

You can pass in either a dict, or a sequence of two-value tuples.

And OrderedDict is trivially converted to such a sequence:

r = requests.post("https://instructure-uploads.s3.amazonaws.com/", files=payload.items())

However, because the collections.OrderedDict() type is a subclass of dict, calling items() is exactly what requests does under the hood, so passing in an OrderedDict instance directly Just Works too.

As such, something else is wrong. You can verify what is being posted by posting to http://httpbin/post instead:

import pprint
pprint.pprint(requests.post("http://httpbin.org/post", files=payload.items()).json())

Unfortunately, httpbin.org does not preserve ordering. Alternatively, you can create a dedicated HTTP post bin at http://requestb.in/ as well; it'll tell you in more detail what goes on.

Using requestb.in, and by replacing '@log.txt' with an open file object, the POST from requests is logged as:

POST /tlrsd2tl HTTP/1.1
User-Agent: python-requests/1.1.0 CPython/2.7.3 Darwin/11.4.2
Host: requestb.in
Content-Type: multipart/form-data; boundary=7b12bf345d0744b6b7e66c7890214311
Content-Length: 1601
Connection: close
Accept-Encoding: gzip, deflate, compress
Accept: */*

--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="content-type"; filename="content-type"
Content-Type: application/octet-stream

text/plain
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="success_action_redirect"; filename="success_action_redirect"
Content-Type: application/octet-stream

https://ian.test.instructure.com/api/v1/files/30652543/create_success?uuid=<opaque_string>
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="Signature"; filename="Signature"
Content-Type: application/octet-stream

<opaque_string>
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="Filename"; filename="Filename"
Content-Type: application/octet-stream


--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="acl"; filename="acl"
Content-Type: application/octet-stream

private
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="Policy"; filename="Policy"
Content-Type: application/octet-stream

<opaque_string>
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="key"; filename="key"
Content-Type: application/octet-stream

account_95298/attachments/30652543/log.txt
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="AWSAccessKeyId"; filename="AWSAccessKeyId"
Content-Type: application/octet-stream

<opaque_string>
--7b12bf345d0744b6b7e66c7890214311
Content-Disposition: form-data; name="file"; filename="log.txt"
Content-Type: text/plain

some
data

--7b12bf345d0744b6b7e66c7890214311--

showing that ordering is preserved correctly.

Note that requests does not support the Curl-specific @filename syntax; instead, pass in an open file object:

 'file': open('log.txt', 'rb')

You may also want to set the content-type field to use title case: 'Content-Type': ...

If you still get a 500 response, check the r.text response text to see what Amazon thinks is wrong.