How to create bulk JSON for Python requests payload?

Question

I have a list of items that I want to add to the payload for a bulk API requests. I don't seem to be able to add the items to the payload JSON the right way.

This is what I want to do:

import json

fruits = ['apple','pear','banana']

headers = {
   'X-Api-Key': API_KEY,
   'Content-Type': 'application/json',
}
 
data = {
   "requests": [
       {
           "params": {
               "fruit": ["apple"]
           }
       },
       {
           "params": {
               "fruit": ["pear"]
           }
       }
       {
           "params": {
               "fruit": ["banana"]
           }
       }
   ]
}
json_responses = requests.post(
   'https://foo.bar/api/bulk',
   headers=headers,
   json=data
).json()

How do I add a new params item to the data payload for each item in my list?

I found this answer from a few years ago but if I try:

my_fruits = json.dumps([dict(fruit=f) for f in fruits])

And then insert it like this I get the error "Object of type set is not JSON serializable".

I'm sure it's because I'm doing something wrong when converting my list to a dictionary and then adding it wrongly but I'm a bit stuck.

EDIT: To clarify, the issue I'm stuck with is how to iterate through the list of fruits and automatically add a param for them in the data dictionary before the request is made.

Sorry - I didn't make this clear. I created the above code manually. I want to loop through a list of hundreds of items in a list and create a new `params` entry in `data` for every item in the list. I'm not sure how to do it automatically. — ipconfuse, Aug 16 '21 at 15:47

score 1 · Accepted Answer · answered Aug 16 '21 at 15:59

The params are in a list. So to add another one you need to append to that list.

paramsList = data.get("requests")

for item in fruits:
  paramsList.append(
    {
       "params" : {
          "fruit" : item,
        }
    }
  )

This should work. It loops through the fruits list and adds them to the requests list in the correct format.

score 1 · Answer 2 · answered Aug 16 '21 at 16:02

It seems to me that you only need a for loop to solve your problem. If you have problems formatting your data with json.dumps, just make your own dictionary and repeat the process for each "fruit".

fruits   = ["a","b"]
requests = []

for f in fruits:
    requests.append({
        "params": { "fruit": [f] }
    })

data = { "requests": requests }

With these lines i got the following result when printing data: {'requests': [{'params': {'fruit': ['a']}}, {'params': {'fruit': ['b']}}]}

Hope it solves the issue!

Thank you Lucas - this was just what I needed. I knew it would be something obvious but I didn't see what. — ipconfuse, Aug 16 '21 at 16:07

How to create bulk JSON for Python requests payload?

2 Answers2