Python Requests: Post JSON and file in single request

Question

I need to do a API call to upload a file along with a JSON string with details about the file.

I am trying to use the python requests lib to do this:

import requests

info = {
    'var1' : 'this',
    'var2'  : 'that',
}

data = json.dumps({
    'token' : auth_token,
    'info'  : info,
})

headers = {'Content-type': 'multipart/form-data'}

files = {'document': open('file_name.pdf', 'rb')}

r = requests.post(url, files=files, data=data, headers=headers)

This throws the following error:

    raise ValueError("Data must not be a string.")
 ValueError: Data must not be a string

If I remove the 'files' from the request, it works.
If I remove the 'data' from the request, it works.
If I do not encode data as JSON it works.

For this reason I think the error is to do with sending JSON data and files in the same request.

Any ideas on how to get this working?

score 38 · Answer 1 · edited Jul 31 '20 at 01:02

38

See this thread How to send JSON as part of multipart POST-request

Do not set the Content-type header yourself, leave that to pyrequests to generate

def send_request():
    payload = {"param_1": "value_1", "param_2": "value_2"}
    files = {
        'json': (None, json.dumps(payload), 'application/json'),
        'file': (os.path.basename(file), open(file, 'rb'), 'application/octet-stream')
    }

    r = requests.post(url, files=files)
    print(r.content)

edited Jul 31 '20 at 01:02

morpheuz

91
1
10

answered Mar 11 '16 at 18:04

ralf htp

9,149
4
22
34

2

let's say Flask is the receiver what would be the way to code flask for that ? – MouIdri Dec 06 '17 at 16:59
1

@Mouldri try `response.data` – Dan Salo Jan 30 '18 at 20:03
3

@MouIdri it would be ```request.form['json']``` – etlds Mar 26 '19 at 16:18
@DanSalo thanks, i forgot to answer, I did find it a will. thanks for your confirmation ( upvoted your comments ) – MouIdri Mar 27 '19 at 15:53
you're the real mvp ! this tricks works for nested data object, that's great ! thanks – Brice Jul 15 '20 at 14:04
2

this one worked for me, but how do we send multiple file in this ? – Maverick Nov 17 '20 at 22:39

proteneer · Accepted Answer · 2013-11-26T07:26:03.213

26

Don't encode using json.

import requests

info = {
    'var1' : 'this',
    'var2'  : 'that',
}

data = {
    'token' : auth_token,
    'info'  : info,
}

headers = {'Content-type': 'multipart/form-data'}

files = {'document': open('file_name.pdf', 'rb')}

r = requests.post(url, files=files, data=data, headers=headers)

Note that this may not necessarily be what you want, as it will become another form-data section.

edited Nov 26 '13 at 07:26

answered Nov 26 '13 at 07:20

proteneer

572
5
17

If I do as you suggest, I get another exception: "need more than 1 value to unpack" and wonder what to do with it :-( – Arkady Apr 09 '15 at 20:03
8

This will only work if `data` are simple key-value pairs (form parameters-like), but all nested stuff will be truncated as HTTP form-encoding is not capable of representing nested data structures. – hoefling Apr 23 '18 at 09:23
Thanks, @hoefling. You've saved my life. I was trying to understand for 1 hour why the hell that library truncates it (or what was happening). – user565447 Oct 12 '18 at 15:25
2

@proteneer This does not seems to be working for me, Doing a call like . `d = requests.post('http://localhost:18090/upload',files={ 'face_image': ('mama_justkilled_aman.jpg', open('mama_justkilled_aman.jpg', 'rb'), 'image/jpeg' ) }, data={ 'gffp': 42 })` but data is not being received by the server – Bala Krishna Feb 13 '19 at 12:48
@hoefling, thanks for your remark. the comment here was working for nested stuff in my case: https://stackoverflow.com/a/35946962/2254425 – Brice Jul 15 '20 at 14:06
2

@BalaKrishna isn't working for me too, server doesn't see data :( – Fedir Alifirenko Apr 06 '21 at 07:35

AChampion · Answer 3 · 2013-10-18T01:41:51.237

3

I'm don't think you can send both data and files in a multipart encoded file, so you need to make your data a "file" too:

files = {
    'data' : data,
    'document': open('file_name.pdf', 'rb')
}

r = requests.post(url, files=files, headers=headers)

edited Oct 18 '13 at 01:41

answered Oct 18 '13 at 01:28

AChampion

29,683
4
59
75

1

How would you decode that? Client would get a python dict not JSON right? it's a question! – ivansabik Sep 25 '16 at 00:16
@sabik: requests encodes the dictionary as form data. – RemcoGerlich Oct 04 '16 at 08:51
1

To note: on the receiving end: the ```request.files['data']``` is a fileStorage tuple. What needs to be done is do a ```request.files['data'].read()``` to get the actual data (which is a json-encoded string) so you'll need to do something like ```json.loads(request.files['data'].read())``` – ewokx Aug 20 '21 at 03:35

score 2 · Answer 4 · answered Apr 10 '20 at 12:43

I have been using requests==2.22.0

For me , the below code worked.

import requests


data = {
    'var1': 'this',
    'var2': 'that'
}

r = requests.post("http://api.example.com/v1/api/some/",
    files={'document': open('doocument.pdf', 'rb')},
    data=data,
    headers={"Authorization": "Token jfhgfgsdadhfghfgvgjhN"}. #since I had to authenticate for the same
)

print (r.json())

score 1 · Answer 5 · answered Nov 29 '18 at 17:16

For sending Facebook Messenger API, I changed all the payload dictionary values to be strings. Then, I can pass the payload as data parameter.

import requests

ACCESS_TOKEN = ''

url = 'https://graph.facebook.com/v2.6/me/messages'
payload = {
        'access_token' : ACCESS_TOKEN,
        'messaging_type' : "UPDATE",
        'recipient' : '{"id":"1111111111111"}',
        'message' : '{"attachment":{"type":"image", "payload":{"is_reusable":true}}}',
}
files = {'filedata': (file, open(file, 'rb'), 'image/png')}
r = requests.post(url, files=files, data=payload)

score 1 · Answer 6 · answered Nov 11 '22 at 12:07

1. Sending request

import json
import requests

cover   = 'superneat.jpg'
payload = {'title': 'The 100 (2014)', 'episodes': json.dumps(_episodes)}
files   = [
            ('json', ('payload.json', json.dumps(payload), 'application/json')),
            ('cover', (cover, open(cover, 'rb')))
          ]
r       = requests.post("https://superneatech.com/store/series", files=files)

print(r.text)

2. Receiving request

You will receive the JSON data as a file, get the content and continue...

Reference: View Here

score -1 · Answer 7 · answered Nov 28 '16 at 12:47

What is more:

files = {
    'document': open('file_name.pdf', 'rb')
}

That will only work if your file is at the same directory where your script is.

If you want to append file from different directory you should do:

files = {
    'document': open(os.path.join(dir_path, 'file_name.pdf'), 'rb')
}

Where dir_path is a directory with your 'file_name.pdf' file.

But what if you'd like to send multiple PDFs ?

You can simply make a custom function to return a list of files you need (in your case that can be only those with .pdf extension). That also includes files in subdirectories (search for files recursively):

def prepare_pdfs():
    return sorted([os.path.join(root, filename) for root, dirnames, filenames in os.walk(dir_path) for filename in filenames if filename.endswith('.pdf')])

Then you can call it:

my_data = prepare_pdfs()

And with simple loop:

for file in my_data:

    pdf = open(file, 'rb')

    files = {
        'document': pdf
    }

    r = requests.post(url, files=files, ...)

The last code block is terrible. That's a DDoS attack! Don't do that. — dsignr, Dec 12 '19 at 08:38

Python Requests: Post JSON and file in single request

7 Answers7

Linked