More elegant and efficient way to iterate through a nested dict?

Question

There are 2 dicts of params:

fb_config = {
  "page_likes" : {
    "page_id": "1111",
                       "metric": "page_fans",
                       "since": "2019-01-01 00:00:00",
                       "until": "2019-01-01 00:00:00",
                       "date_preset" : "yesterday",
                       "period": "day"},

    "page_impressions" : {"page_id": "1111",
                       "metric": "page_impressions",
                       "since": "yesterday",
                       "until": "yesterday",
                       "date_preset" : "yesterday",
                       "period": "day"},

    "page_engaged_users" : {"page_id": "1111",
                       "metric": "page_engaged_users",
                       "since": "today",
                       "until": "today",
                       "date_preset" : "yesterday",
                       "period": "day"}}

and

ga_config =

{

'view_id_123': {'view_id': '123',
                           'start_date': "2019-01-01 00:00:00",
                           'end_date': "2019-01-01 00:00:00",
                           'metrics': [{'expression': 'ga:sessions'}, {'expression':'ga:pageviews'}, {'expression':'ga:users'}, {'expression':'ga:bounces'},
                          {'expression':'ga:avgSessionDuration'}],
                           'dimensions': [{'name': 'ga:year'}, {'name': 'ga:userType'}, {'name': 'ga:sessionCount'}, {'name': 'ga:browser'},
                                          {'name': 'ga:source'}]},

'view_id_456': {'view_id': '456',
                           'start_date': "today",
                           'end_date': "today",
                           'metrics': [{'expression': 'ga:bounces'}, {'expression':'ga:users'}],
                           'dimensions': [{'name': 'ga:mobileDeviceModel'}, {'name': 'ga:dataSource'}]},

'view_id_789': {'view_id': '789',
                           'start_date': "yesterday",
                           'end_date': "yesterday",
                           'metrics': [{'expression': 'ga:bounces'}, {'expression':'ga:users'}],
                           'dimensions': [{'name': 'ga:mobileDeviceModel'}, {'name': 'ga:dataSource'}]}

}

In the code I can't keep the params and want to pull them from a json file. After loading the file all the dates have to be converted from str to datetime format.

This process is accomlished through several loops:

for values in params.values():
    if 'since' in values:
        if values['since'] == 'today':
            values['since'] = datetime.now()
        elif values['since'] == 'yesterday':
            values['since'] = datetime.now() - timedelta(1)
        else: 
            values['since'] = dt.datetime.strptime(values['since'], '%Y-%m-%d %H:%M:%S')

for values in params.values():
    if 'until' in y:
        if values['until'] == 'today':
            values['until'] = datetime.now()
        elif values['until'] == 'yesterday':
            values['until'] = datetime.now() - timedelta(1)
        else:
            values['until'] = dt.datetime.strptime(y['until'], '%Y-%m-%d %H:%M:%S')

for values in params.values():
    if 'start_date' in values:
        if values['start_date'] == 'today':
            values['start_date'] = datetime.now()
        elif values['start_date'] == 'yesterday':
            values['start_date'] = datetime.now() - timedelta(1)
        else:
            values['start_date'] = dt.datetime.strptime(values['start_date'], '%Y-%m-%d %H:%M:%S')

for values in params.values():
    if 'end_date' in values:
        if values['end_date'] == 'today':
            values['end_date'] = datetime.now()
        elif values['end_date'] == 'yesterday':
            values['end_date'] = datetime.now() - timedelta(1)
        else:
            values['end_date'] = dt.datetime.strptime(values['end_date'], '%Y-%m-%d %H:%M:%S')

But the code is really ugly and DRY is violated. Are there better ways to iterate through nested dics and change the values of the certain keys?

score 1 · Accepted Answer · answered Oct 01 '19 at 13:21

This is a bit more compact:

for values in params.values():
    for k in ('since','until','start_date','end_date'):
        if k in values:
            if values[k] == 'today':
                values[k] = datetime.now()
            elif values[k] == 'yesterday':
                values[k] = datetime.now() - timedelta(1)
            else: 
                values[k] = dt.datetime.strptime(values[k], '%Y-%m-%d %H:%M:%S')

score 1 · Answer 2 · answered Oct 01 '19 at 13:34

I am not a Python developer, but I can see that you are looping 4 times over the same data, when you should do it only once:

def formatDate(value, default):
    if value == 'today':
        date = datetime.now()
    elif value == 'yesterday':
        date = datetime.now() - timedelta(1)
    else:
        date = default

    return date

def your_function():

    for values in params.values():
        if 'since' in values:
            values['since'] = formatDate(values['since'], dt.datetime.strptime(values['since'], '%Y-%m-%d %H:%M:%S'))

        if 'until' in values:
            values['until'] = formatDate(values['until'], dt.datetime.strptime(y['until'], '%Y-%m-%d %H:%M:%S'))

        if 'start_date' in values:
            values['start_date'] = formatDate(values['start_date'], dt.datetime.strptime(values['start_date'], '%Y-%m-%d %H:%M:%S'))

        if 'end_date' in values:
            values['end_date'] = formatDate(values['end_date'], dt.datetime.strptime(values['end_date'], '%Y-%m-%d %H:%M:%S'))

Code not tested, but I would extract the common logic to a function and have only one foreach loop.

score 1 · Answer 3 · answered Oct 01 '19 at 13:42

Instead of iterating over a nested dict after you load json, you need to use object_hook method of json.loads() to deserialize dates on import (I assume you're using built-in json library to parse json).

This has already been answered, please have a look at this thread: How to convert to a Python datetime object with JSON.loads?

score 1 · Answer 4 · answered Oct 01 '19 at 14:06

Consider a nested dictionary comprehension and a dict mapping function for time conversion:

def time_conversion(arg):
    switcher = {
        'today': datetime.now(),
        'yesterday': datetime.now() - timedelta(1)
    }

    output = (datetime.strptime(arg, '%Y-%m-%d %H:%M:%S') 
                  if arg not in switcher.keys() else switcher.get(arg))
    return output


new_params = {outer_k:{inner_k: time_conversion(inner_v) 
                          if inner_k in ('since', 'until', 'date_preset') else inner_v 
                       for inner_k, inner_v in outer_v.items()} 
              for outer_k, outer_v in params.items()}

More elegant and efficient way to iterate through a nested dict?

4 Answers4