Copy nested dictionary to another dictionary with all the levels

Question

For some third party APIs, there is a huge data that needs to be sent in the API parameters. And input data comes to our application in the CSV format.

I receive all the rows of the CSV containing around 120 columns, in a plane dict format by CSV DictReader.

file_data_obj = csv.DictReader(open(file_path,  'rU'))

This gives me each row in following format:

CSV_PARAMS = {
    'param7': "Param name",
    'param6': ["some name"],
    'param5': 1234,
    'param4': 999999999,
    'param3': "some ",
    'param2': {"x name":"y_value"},
    'param1': None,
    'paramA': "",
    'paramZ': 2.687
}

And there is one nested dictionary containing all the third-party API parameters as keys with blank value.

eg. API_PARAMS = {
    "param1": "",
    "param2": "",
    "param3": "",
    "paramAZ": {
        "paramA": "",
        "paramZ": {"test1":1234, "name":{"hello":1}},
        ...
    },
    "param67": {
        "param6": "",
        "param7": ""
    },
    ...
  }

I have to map all the CSV Values to API parameters dynamically. following code works but upto 3 level nesting only.

def update_nested_params(self, paramdict, inpdict, result={}):
    """Iterate over nested dictionary up to level 3 """
    for k, v in paramdict.items():
        if isinstance(v, dict):
            for k1, v1 in v.items():
                if isinstance(v1, dict):
                    for k2, _ in v1.items():
                        result.update({k:{k1:{k2: inpdict.get(k2, '')}}})
                else:
                    result.update({k:{k1: inpdict.get(k1, '')}})
        else:
            result.update({k: inpdict.get(k, '')})
    return result




self.update_nested_params(API_PARAMS, CSV_PARAMS)

Is there any other efficient way to achieve this for n number of nestings of the API Parameters?

unrelated: setting the default value of result to `{}` will probably give you unintended side effects if you call it more than once ... — donkopotamus, Aug 08 '16 at 07:21
You *really* don't want to use `result={}` as a default keyword argument here, see ["Least Astonishment" in Python: The Mutable Default Argument](http://stackoverflow.com/q/1132941) — Martijn Pieters, Aug 08 '16 at 07:42

Martijn Pieters · Accepted Answer · 2019-08-06T09:51:10.683

You could use recursion:

def update_nested_params(self, template, source):
    result = {}
    for key, value in template.items():
        if key in source:
            result[key] = source[key]
        elif not isinstance(value, dict):
            # assume the template value is a default
            result[key] = value
        else:
            # recurse
            result[key] = self.update_nested_params(value, source)
    return result

This copies the 'template' (API_PARAMS) recursively, taking any key it finds from source if available, and recurses if not but the value in template is another dictionary. This handles nesting up to sys.getrecursionlimit() levels (default 1000).

Alternatively, use an explicit stack:

# extra import to add at the module top
from collections import deque

def update_nested_params(self, template, source):
    top_level = {}
    stack = deque([(top_level, template)])
    while stack:
        result, template = stack.pop()
        for key, value in template.items():
            if key in source:
                result[key] = source[key]
            elif not isinstance(value, dict):
                # assume the template value is a default
                result[key] = value
            else:
                # push nested dict into the stack
                result[key] = {}
                stack.append((result[key], value))
    return top_level

This essentially just moves the call stack used in recursion to an explicit stack. The order in which keys are processed changes from depth to breath first but this doesn’t matter for your specific problem.

Copy nested dictionary to another dictionary with all the levels

1 Answers1