Using Python Requests to send file and JSON in single request

Question

I'm trying to POST to an API (Build using SlimPHP) which accepts an image along with additional image meta data in the form of JSON.

I've verified the API works correctly using a REST client tool and can successfully POST to the service. All data is stored correctly.

I'm now trying to POST using Python - however my JSON data doesn't appear to be saving.

My code:

    data = {'key1': 'value1', 'key2': 'value2'}
    url = 'http://mydomain.com/api/endpoint'
    headers = {'Authorization': 'my-api-key'}
    files = {'file': (FILE, open(PATH, 'rb'), 'image/jpg', {'Expires': '0'})}
    r = requests.post(url, files=files, headers=headers, data=data)

--

I've attempted to set additional headers,

ie:/

headers = {'Authorization': 'unique-auth-key', 'Content-type': 'multipart/form-data'}

or

headers = {'Authorization': 'unique-auth-key', 'Content-type': 'application/json'}

These result in a 500 error.

UPDATE 14/07/2014:

Using a chrome extension (Advanced Rest Client) my POST is successful - here's what the console shows as the payload:

------WebKitFormBoundarysBpiwrA3hnGPUbMA
Content-Disposition: form-data; name="data"
test
------WebKitFormBoundarysBpiwrA3hnGPUbMA
Content-Disposition: form-data; name="file"; filename="image.jpg"
Content-Type: image/jpeg
------WebKitFormBoundarysBpiwrA3hnGPUbMA--

I'm not quite sure what this signifies...

You have to provide API specification for the call to run. A sample using `curl`, HTTPie or similir tool would help. Usually, the JSON data are sent in body as the only content. If you want to combine it with file upload, you are probably going to use sort of form, with one file being the image, and another text field containing the JSON. Anyway, `data` shall be a string, you provide a dictionary. Changing it to `data=json.dumps(data)` could help. — Jan Vlcinsky, Jul 08 '14 at 22:02
This cURL request works: curl -i -F data='{"key": "value", "key": "value", "key": "value"}' -F name=image.jpg -F file=@image.jpg -H "Authorization: abcd-efgh-ijkl-mnop" http://www.my-url.com/api/endpoint — Simon, Jul 14 '14 at 13:44

Jeremy Allen · Accepted Answer · 2014-07-15T01:33:06.710

22

Your problem is that you are using the image metadata as the source of key/value pairs to be posted. Instead of sending it as the value of one of those key/value pairs.

The following code will send a request much like the curl statement you provided:

url = 'my-url.com/api/endpoint'
headers = {'Authorization': 'my-api-key'}
image_metadata = {'key1': 'value1', 'key2': 'value2'}
data = {'name': 'image.jpg', 'data': json.dumps(image_metadata)}
files = {'file': (FILE, open(PATH, 'rb'), 'image/jpg', {'Expires': '0'})}
r = requests.post(url, files=files, headers=headers, data=data)

edited Jul 15 '14 at 01:33

answered Jul 15 '14 at 01:23

Jeremy Allen

6,434
2
26
31

As much as I'd like the bounty, after investigating this, I concur. After looking at http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file and posting test requests to 'http://httpbin.org/post', I replicated the curl output using python nearly identical to this answer. (Two caveats: the image_metadata dict obviously can't support multiple instances of 'key':'value', and FILE I guess is the filename.) – John Hazen Jul 16 '14 at 04:57
2

Hi Jeremy. Can you explain please what the FILE variable stands for? Trying to use similar solution for my code – Pavel Zagalsky Aug 07 '17 at 04:37

score 0 · Answer 2 · answered May 24 '20 at 10:12

If you need to post nested JSON data and send files in single request, you need to use json.dumps for these elements (lists and mappings in the example below)

import requests

session = requests.Session()
session.post(login_url, json=your_credentials_dict)  # auth , not required
mapping = {"email": "Email", "First Name": "first_name"}
post_data = dict(
   mode=1,
   lists=json.dumps(["133", "899", "911"]),
   mapping=json.dumps(mapping),
)
files_data = {'file': ('your-file-name.csv', file_bytes_content)}
response = session.post(<your-url>, data=post_data, files=files_data)

And as a result will be sent request like this:

********** post file data with mapping **********
 --> REQUEST: POST /v3/contacts/import/file/ HTTP/1.1
 --> REQUEST: Host: <your-host>:4000
 --> REQUEST: User-Agent: python-requests/2.23.0
 --> REQUEST: Accept-Encoding: gzip, deflate
 --> REQUEST: Accept: */*
 --> REQUEST: Connection: keep-alive
 --> REQUEST: Cookie: jwt=eyJ0e***
 --> REQUEST: Content-Length: 654
 --> REQUEST: Content-Type: multipart/form-data; boundary=db850c3230988e010b1ebe21be3fb344
 --> REQUEST: 
 --> REQUEST: --db850c3230988e010b1ebe21be3fb344
Content-Disposition: form-data; name="lists"

["7syewz9qUz0CbaqhaGm", "LBfWKJwq4Q"]
--db850c3230988e010b1ebe21be3fb344
Content-Disposition: form-data; name="mode"

2
--db850c3230988e010b1ebe21be3fb344
Content-Disposition: form-data; name="mapping"

{"email": "Email", "First Name": "first_name"}
--db850c3230988e010b1ebe21be3fb344
Content-Disposition: form-data; name="file"; filename="good.csv"

first_name;last_name;email;age
John;Doe;test.testson_1@test.com;50
Mark;Twen;test.testson_2@test.com;20

--db850c3230988e010b1ebe21be3fb344--

requests-toolbelt package was used to generate the log above https://pypi.org/project/requests-toolbelt/

Using Python Requests to send file and JSON in single request

2 Answers2

Linked