combine values in a list of dicts containing a list of dicts

Question

This is way out of my comfort range and I'm not even sure I can describe it well enough. I have a file that is a list of dicts that contain another list of dicts. An excerpt of the data structure is below:

j_traffic =
[   
    {
    "timePeriod":   "2017-08-04T15:20:00.000+0000",
    "applicationTrafficPerApplication": [
         {
         "applicationId":   39,
         "applicationName": "HTTP",
         "trafficInboundBps":   148760,
         "trafficOutboundBps":  5673493,
         "trafficWithinBps":    0
         },
         {
         "applicationId":   41,
         "applicationName": "HTTPS",
         "trafficInboundBps":   16805,
         "trafficOutboundBps":  546937,
         "trafficWithinBps":    0
         }
         ]
    },
    {
    "timePeriod":   "2017-08-04T15:15:00.000+0000",
    "applicationTrafficPerApplication": [
         {
         "applicationId":   39,
         "applicationName": "HTTP",
         "trafficInboundBps":   157569,
         "trafficOutboundBps":  5769206,
         "trafficWithinBps":    0
         },
         {
         "applicationId":   41,
         "applicationName": "HTTPS",
         "trafficInboundBps":   17454,
         "trafficOutboundBps":  590421,
         "trafficWithinBps":    0
         },
         {
         "applicationId":   44,
         "applicationName": "DNS",
         "trafficInboundBps":   18218,
         "trafficOutboundBps":  13683,
         "trafficWithinBps":    0
         },
         {
         "applicationId":   45,
         "applicationName": "SNMP",
         "trafficInboundBps":   14,
         "trafficOutboundBps":  0,
         "trafficWithinBps":    0
         }  
         ]
    },  
    {   
    "timePeriod":   "2017-08-04T15:05:00.000+0000",
    "applicationTrafficPerApplication": [
         {  
         "applicationId":   39,
         "applicationName": "HTTP",
         "trafficInboundBps":   139897,
         "trafficOutboundBps":  5073320,
         "trafficWithinBps":    0
         }, 
         {  
         "applicationId":   41,
         "applicationName": "HTTPS",
         "trafficInboundBps":   22592,
         "trafficOutboundBps":  457962,
         "trafficWithinBps":    0
         }, 
         {  
         "applicationId":   44,
         "applicationName": "DNS",
         "trafficInboundBps":   19903,
         "trafficOutboundBps":  14033,
         "trafficWithinBps":    0
         }
         ]
    }
]

I am trying to understand how I can create a new dict using "applicationName" value as the keys and the values are the sum of all values of key "trafficInboundBps" that would look like this:

inboundTraffic = {"HTTP": 446316, "HTTPS": 56581, "DNS": 38121, "SNMP": 14}

I've tried suggestions I found on here but can't wrap my head around how to parse the nested levels with the following: inboundTraffic = dict.fromkeys(set().union(*j_traffic))

Any takers?

thanks!

score 0 · Answer 1 · answered Aug 24 '17 at 22:34

This is one possibility to do what you're asking:

# Extract outer dictionaries as a list
lst = [s["applicationTrafficPerApplication"] for s in j_traffic]

# Turn first element of lst into a dictionary 
inboundTraffic={s2["applicationName"]: s2["trafficInboundBps"] for s2 in lst[0]}
# Process remaining elements - combine and add
for comp in lst[1:]:
    temp = {s2["applicationName"]: s2["trafficInboundBps"] for s2 in comp}
    # This turns both dictionaries into sets, selects all elements 
    # (I assume that's why it's using sets - to have access to all), 
    # then adds the resepective elements - 0 in .get(k,0) signifies that
    # "0" will be added if particular element doesn't exist in the second set/dictionary
    inboundTraffic = {k: inboundTraffic.get(k,0) + temp.get(k,0) for k in set(inboundTraffic) | set(temp)} 

print inboundTraffic

I'm still learning the pythonic ways of doing things so I bet there is a shorter and more proper solution - but this indeed does the trick.

The last line in the for loop was due to this post Merge and sum of two dictionaries.

Do you want your output sorted?

Anupam Khulbe · Accepted Answer · 2017-08-25T07:33:02.600

Here's a plain and simple code for processing the j_traffic list and getting the expected output.

output = dict()
# iterate over the list of outer dicts
for outer_dict in j_traffic:
    # grab the list assigned to key applicationTrafficPerApplication
    application_traffic_per_application = outer_dict['applicationTrafficPerApplication']
    # iterate over the list of inner dicts
    for inner_dict in application_traffic_per_application:
        application_name = inner_dict['applicationName']
        traffic_inbound_bps = inner_dict['trafficInboundBps']
        if application_name in output:
            output[application_name] += int(traffic_inbound_bps)
        else:
            output[application_name] = int(traffic_inbound_bps)

print(output)

combine values in a list of dicts containing a list of dicts

2 Answers2