How to post dicts with lists using requests in Python?

Question

I need to send nested request. Here is example of dict:

data = {
        "customer_id": "customer_1",
        "time": get_timestamp(),
        "data_1": ["1", "2"],
        "data_2": ["6", "7"],
    }

There can be larger amount of elements in lists and more lists, this is just an example. When sending r = requests.post(url, data=data) or r = requests.post(url, json=data) or r = requests.post(url, data=json.dumps(data))

I get a validation message that there a no required fields and I checked that data hasn't been sent. However, if I send it like this it works:

data = {
        "customer_id": "customer_1",
        "time": get_timestamp(),
        "data_1[0]": "1",
        "data_1[1]": "2",
        "data_2[0]": "6",
        "data_2[1]": "7"
    }

Is there a way to send lists using requests without having to put every element as different key?

Edit. I forgot to mention that I need to send files together too. At first I hasn't thought that files can be a problem, but they of course can (I wanted to simplify question as much as possible). I add files: files = {"files": open(path_to_file, "rb")} and send adding files=files to POST request. So the whole code would look like this:

url = ""
token = "token"
headers = {"Authorization": "Bearer " + token}
data = {
    "customer_id": "customer_1",
    "time": get_timestamp(),
    "data_1": ["1", "2"],
    "data_2": ["6", "7"],
}
path_to_file = "file.jpg"
files = {"files": open(path_to_file, "rb")}
r = requests.post(url, data=data, files=files, headers=headers, verify=False)

I have a postman specification for that:

Can you share the error message ? Because your try n°2 and n°3 are correct. Do you have a definition of what i expected, or the server code ? — azro, May 29 '20 at 08:55
ug, it looks like it is reading your json as a string and then evaluating it, try json=son.dumps(data) with your first dict, see what happend — E.Serra, May 29 '20 at 08:56
https://stackoverflow.com/questions/38380086/sending-list-of-dicts-as-value-of-dict-with-requests-post-going-wrong Please refer to this. — Shafay, May 29 '20 at 08:57
2nd try. Error is "The customer_id field is required." and it's repeated for all the keys. Same is for the 3rd try (using json=data). — py_ml, May 29 '20 at 09:00
As for that stackoverflow question, there are jsons sent in that answer, but in my situation server expects it to be in array. — py_ml, May 29 '20 at 09:01
@E.Serra I tried that, same as without json.dumps, keeps represented as string — py_ml, May 29 '20 at 09:15
what does the `r = requests.options(url)` return? can you please let us know? is it possible to show us the url so can try at our end? — Tarique, May 30 '20 at 04:26
Without sharing more about the actual server processing the requests and some actual code, it's going to be very hard to say anything meaningful. If something like `"data_1[0]": "1"` in the dictionary works, it would appear that your server is expecting more than just basic json, unless you are leaving out some funky code in the middle. — Grismar, Jun 01 '20 at 05:30
@Grismar I have updated the question. The actual server processing isn't done by me and I have only that spec, not the code. — py_ml, Jun 01 '20 at 08:56

score 0 · Accepted Answer · answered Jun 02 '20 at 04:39

If your server requires you to include values in a list as individual items with an index (i.e. "data_1[0]": "1" and "data_1[1]": "2" instead of "data_1":["1", "2"]), then there's really nothing anyone here can tell you that will change that. Whether that's actually true is not something anyone here can validate if you don't share more information about the server or the protocol.

Of course it's pretty easy to map the data structure you'd like onto the data structure the server needs:

def get_what_you_need(data, unpack_lists):
    result = dict(data)
    for unpack_list in unpack_lists:
        assert unpack_list in data
        for key, value in enumerate(data[unpack_list]):
            result[f'{unpack_list}[{key}]'] = value
        del result[unpack_list]
    return result


you_like = {
    "customer_id": "customer_1",
    "time": get_timestamp(),
    "data_1": ["1", "2"],
    "data_2": ["6", "7"],
}

you_need = get_what_you_need(you_like, ['data_1', 'data_2'])
print(you_need)

As for files, it doesn't appear you actually have a question there and it doesn't have any bearing on the above.

Thanks, I mark it as accepted as this idea is more universal for any possible depth of request. However, I found that python requests parse "one layer" nested arrays well after adding brackets to dict name, for example, in this situation `"data_1[]": ["1", "2"],` — py_ml, Jun 03 '20 at 19:03
That's an interesting addition, though that would appear to have more to do with the server processing the json sent, than anything on the side of the sender. Even if the server was written in Python, that's not a given - but clearly it works in your case. — Grismar, Jun 03 '20 at 23:04

How to post dicts with lists using requests in Python?

1 Answers1