Extract & Append Inner Dictionary to Outer Dictionary

Question

Overview

For converting API responses to Pandas Dataframes I was looking for a solution that would extract Inner Dictionaries from the Outer(main) Dictionary and append the key value pairs from the Inner Dictionary onto the Outer Dictionary to have one nice Dictionary that could be converted to a Dataframe. I was unable to find a solution so I built the solution my self with the help from another question.

If this has been answered already or there is an issue with my solution please let me know. I hope this helps someone as I spent many hours attempting to find a solution.

Input

dict1 = {'outside_dict': {'middle_dict1': {'innerDict': 1}, 'middle_dict2' : 2},'innerDict': 3}

dict2 = {'outside_dict': {'middle_dict1': {'innerDict': 1}, },}

Output

dict1 = {'innerDict': 3, 'middle_dict2_outside_dict': 2, 'innerDict_middle_dict1_outside_dict': 1}

dict2 = {'innerDict_middle_dict1_outside_dict': 1}

Answer 1

Explanation

The merge_dicts function comes from @Michael Spector answer on the question How to merge dictionaries of dictionaries?

The merge_dicts function merges two Dictionaries together and overrides any conflicting keys. It also appends the InnerDict key to the new OuterDict key for reference of where is came from.

The extract_innerDict_to_outerDict function goes through a copy of the items in the OuterDict to check if any values of the Outer Dictionary are Dictionaries themselves. If so it will call itself again to dive deeper into the OuterDict until there is no more Dictionaries to be found. Once the extraction is done it delete the key which has a Dictionary type as the value and proceed to the next key.

Code

def extract_innerDict_to_outerDict(mainDict):
  """
  This function extracts innerDicts from outerDicts and appends to outerDict data.
  """
  for key, value in mainDict.copy().items():
      if isinstance(value, dict):
        extract_innerDict_to_outerDict(mainDict[key])
        merge_dicts(mainDict, mainDict[key], key)
        del mainDict[key]
  return mainDict

def merge_dicts(dict1, dict2, innerDict_key):
  """ Recursively merges dict2 into dict1 """
  if not isinstance(dict1, dict) or not isinstance(dict2, dict):
      return dict2
  for k in dict2:
      if k in dict1:
          dict1[k + '_' + innerDict_key] = merge_dicts(dict1[k], dict2[k])
          print
      else:
          dict1[k + '_' + innerDict_key] = dict2[k]
  return dict1

Real World Application

Here is an example of a response from the Github Collaborators API. You get the permissions data in a nested dictionary.

Input

githubexample = { "login": "octocat", "id": 1, "node_id": "MDQ6VXNlcjE=", "avatar_url": "https://github.com/images/error/octocat_happy.gif", "gravatar_id": "", "url": "https://api.github.com/users/octocat", "html_url": "https://github.com/octocat", "followers_url": "https://api.github.com/users/octocat/followers", "following_url": "https://api.github.com/users/octocat/following{/other_user}", "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}", "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/octocat/subscriptions", "organizations_url": "https://api.github.com/users/octocat/orgs", "repos_url": "https://api.github.com/users/octocat/repos", "events_url": "https://api.github.com/users/octocat/events{/privacy}", "received_events_url": "https://api.github.com/users/octocat/received_events", "type": "User", "site_admin": False, "permissions": { "pull": True, "triage": True, "push": True, "maintain": True, "admin": True }, "role_name": "write" }

Output

Once run through the code the nested permission dictionary is extracted from the nested dictionary and appended to the outer(main) dictionary.

githubexample = { "login": "octocat", "id": 1, "node_id": "MDQ6VXNlcjE=", "avatar_url": "https://github.com/images/error/octocat_happy.gif", "gravatar_id": "", "url": "https://api.github.com/users/octocat", "html_url": "https://github.com/octocat", "followers_url": "https://api.github.com/users/octocat/followers", "following_url": "https://api.github.com/users/octocat/following{/other_user}", "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}", "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/octocat/subscriptions", "organizations_url": "https://api.github.com/users/octocat/orgs", "repos_url": "https://api.github.com/users/octocat/repos", "events_url": "https://api.github.com/users/octocat/events{/privacy}", "received_events_url": "https://api.github.com/users/octocat/received_events", "type": "User", "site_admin": false, "role_name": "write", "pull_permissions": true, "triage_permissions": true, "push_permissions": true, "maintain_permissions": true, "admin_permissions": true }