How to automatically parse JSON and get all key value pairs in a variable?

Question

I'm trying to dynamically create a database without hard coding all of the API response values for columns and rows so I want to learn how to automatically parse through JSON and put all the keys/values in a variable so I can sort through and put them in the db.

Lets say I had some JSON like this:

(modified snippet from destiny 2 API response indented by 5)

{
     "Response": {
          "profile": {
               "data": {
                    "userInfo": {
                         "crossSaveOverride": 1,
                         "applicableMembershipTypes": [
                              3,
                              1
                         ],
                         "isPublic": true,
                         "membershipType": 1,
                         "membershipId": "123",      
                         "displayName": "Name1",
                         "bungieGlobalDisplayName": "name again",
                         "bungieGlobalDisplayNameCode": 0000
                    },
                    "dateLastPlayed": "2021-6-18T02:33:01Z",        
                    "versionsOwned": 127,
                    "characterIds": [
                         "id1",
                         "id2",
                         "id3"
                    ],
                    "seasonHashes": [
                         3243434344,
                         3423434244,
                         3432342443,
                         3434243244
                    ],
                    "currentSeasonHash": 3434243244,
                    "currentSeasonRewardPowerCap": 1330
               },
               "privacy": 1
          }
     },
     "ErrorCode": 1,
     "ThrottleSeconds": 0,
     "ErrorStatus": "Success",
     "Message": "Ok",
     "MessageData": {}
}

I want to automatically parse through and get all of the key/values minus the error code area so everything under "Response". It would all go into the database for example:

displayName	isPublic
Name1	True
Name2	False

I know how to parse normally or loop through but only to grab values like:

displayName = Response["Response"]["profile"]["data"]["userInfo"]["displayName"]

How does one grab all keys and values and then automatically store in a variable from the lowest level? Also, how would one exclude keys if they are not needed?

EDIT: Adding Clarification

I learned that this JSON response type is a dict and I can use Response.keys() and Response.values() to get the keys and values.

What I am asking is, from Response["Response"], how to get all of the keys and values down to the bottom level and exclude the ones I don't need.

For example if I do:

r = Response["Response"] 
for key in r.keys():
    print(key)

I'm only going to get profile which is what I don't need.

I would have to do:

r = Response["Response"]["profile"]["data"]["userInfo"] 
for key in r.keys():
    print(key)

which would return

crossSaveOverride
applicableMembershipTypes
isPublic
membershipType
membershipId
displayName
bungieGlobalDisplayName
bungieGlobalDisplayNameCode

Which is what I need but I do not want to manually define ["Response"]["profile"]["data"]["userInfo"]or similar for every response. I want to grab everything automatically while excluding the items I don't need.

You can try using the json module. `import json` and then use `json.loads` function. — sushant, Oct 24 '21 at 00:36
What exactly do you mean by "get all of the key/values"? I.E. how and in what form? — martineau, Oct 24 '21 at 00:36
@martineau for example, parsing through and getting **displayName** & the value **Name1**. I can get down to the value "Name1" but I don't know how to grab the string "displayname" itself. I believe getting everything in a dict would work, but I do not know. — dekubaka, Oct 24 '21 at 01:01
I'm not sure if I understood your question correctly, but to get all the keys you can use the `response.keys()`. — Gvinfinity, Oct 24 '21 at 01:14
You can convert the JSON into a Python dictionary by using `json.load()` like @sushant suggested, but that doesn't answer my question about, now that you have that dictionary, how you want to get all the key/value pairs in it when you basically already have that in the dictionary — so it's unclear what you mean. — martineau, Oct 24 '21 at 01:14
I think you're looking for something like https://stackoverflow.com/a/39234154/2111837 — iScripters, Oct 24 '21 at 10:21
@iScripters Yes! That is what I'm looking for. Could you post as the answer? — dekubaka, Oct 24 '21 at 16:35
@martineau I did not know that I was already dealing with a dictionary from the beginning. My question should have been how to loop through and get every nested key/value pair in a dictionary & exclude the ones I don't need. — dekubaka, Oct 24 '21 at 17:10
You're still doing it in "how to loop through and get every nested key/value pair" where all you say is "get". What does that mean? — martineau, Oct 24 '21 at 17:13
@martineau get? retrieve? pull? extract? I don't know what I'm missing here. How would you say it? — dekubaka, Oct 24 '21 at 17:22
All those words would also be inadequate. Get the pairs in what way, doing what with them is what's missing for someone to be able answer your question. — martineau, Oct 24 '21 at 17:28
Just explain what "put all the keys/values in a variable so I can sort through and put them in the db" entails. It's unclear what the type of such as "variable" would be. It sounds like it's a container of some kind. — martineau, Oct 24 '21 at 17:55
@martineau Store all keys/values in a variable first from the JSON and then search through it. This is because I did not know it was treated as a dictionary already. The db part did not matter and could have been removed. If the variable type matters then I would have to specify a dictionary variable ? — dekubaka, Oct 24 '21 at 18:54

score 0 · Accepted Answer · answered Oct 24 '21 at 19:11

0

Dmitry Torba over at Get all keys of a nested dictionary posted a function that does what you want.

def recursive_items(dictionary):
    for key, value in dictionary.items():
        if type(value) is dict:
            yield from recursive_items(value)
        else:
            yield (key, value)

a = {'a': {1: {1: 2, 3: 4}, 2: {5: 6}}}

for key, value in recursive_items(a):
    print(key, value)

answered Oct 24 '21 at 19:11

iScripters

403
3
13

How does this eliminate the need to "manually define ["Response"]["profile"]["data"]["userInfo"]"? – martineau Oct 24 '21 at 19:42

score 0 · Answer 2 · answered Dec 17 '22 at 19:16

I have tried this approach, what if there is list type in given dictionary!

CallPretty.py

def JsonPrint(response, response_Keys):
    print()
    print(response_Keys)
    for i in range(len(response_Keys)):

        if type(response[response_Keys[i]]) == list:
            print()
            
            for j in range(len(response[response_Keys[i]])):
                if type(response[response_Keys[i]][j]) == dict:
                    JsonPrint(response[response_Keys[i]][j], list(response[response_Keys[i]][j].keys()))

        elif type(response[response_Keys[i]]) == dict:
            JsonPrint(response[response_Keys[i]], list(response[response_Keys[i]].keys()))

        else:
            print(response_Keys[i], ':')
            for j in range(len(response[response_Keys[i]].split('\n'))):
                print('\t', response[response_Keys[i]].split('\n')[j])

If you have a new question, please ask it by clicking the [Ask Question](https://stackoverflow.com/questions/ask) button. Include a link to this question if it helps provide context. - [From Review](/review/late-answers/33439448) — Edward Ji, Dec 20 '22 at 12:46

How to automatically parse JSON and get all key value pairs in a variable?

EDIT: Adding Clarification

2 Answers2