Looping though a list trying to create an dict

Question

I have a list of data example:

res = { 'results': [
{'consumption': 0.025, 'interval_start': '2021-06-27T00:00:00+01:00', 'interval_end': '2021-06-27T00:30:00+01:00'},
{'consumption': 0.043, 'interval_start': '2021-06-26T23:30:00+01:00', 'interval_end': '2021-06-27T00:00:00+01:00'},
{'consumption': 0.053, 'interval_start': '2021-06-26T23:00:00+01:00', 'interval_end': '2021-06-26T23:30:00+01:00'},
{'consumption': 0.056, 'interval_start': '2021-06-26T22:30:00+01:00', 'interval_end': '2021-06-26T23:00:00+01:00'},
{'consumption': 0.031, 'interval_start': '2021-06-26T22:00:00+01:00', 'interval_end': '2021-06-26T22:30:00+01:00'},
{'consumption': 0.129, 'interval_start': '2021-06-26T21:30:00+01:00', 'interval_end': '2021-06-26T22:00:00+01:00'},
{'consumption': 0.19,  'interval_start': '2021-06-26T21:00:00+01:00', 'interval_end': '2021-06-26T21:30:00+01:00'},
{'consumption': 0.164, 'interval_start': '2021-06-26T20:30:00+01:00', 'interval_end': '2021-06-26T21:00:00+01:00'},
{'consumption': 0.145, 'interval_start': '2021-06-26T20:00:00+01:00', 'interval_end': '2021-06-26T20:30:00+01:00'},
{'consumption': 0.213, 'interval_start': '2021-06-26T19:30:00+01:00', 'interval_end': '2021-06-26T20:00:00+01:00'},
{'consumption': 0.167, 'interval_start': '2021-06-26T19:00:00+01:00', 'interval_end': '2021-06-26T19:30:00+01:00'},
{'consumption': 0.333, 'interval_start': '2021-06-26T18:30:00+01:00', 'interval_end': '2021-06-26T19:00:00+01:00'},
{'consumption': 0.133, 'interval_start': '2021-06-26T18:00:00+01:00', 'interval_end': '2021-06-26T18:30:00+01:00'},
{'consumption': 0.211, 'interval_start': '2021-06-26T17:30:00+01:00', 'interval_end': '2021-06-26T18:00:00+01:00'},
{'consumption': 0.135, 'interval_start': '2021-06-26T17:00:00+01:00', 'interval_end': '2021-06-26T17:30:00+01:00'},
{'consumption': 0.158, 'interval_start': '2021-06-26T16:30:00+01:00', 'interval_end': '2021-06-26T17:00:00+01:00'},
{'consumption': 0.073, 'interval_start': '2021-06-26T16:00:00+01:00', 'interval_end': '2021-06-26T16:30:00+01:00'},
{'consumption': 0.077, 'interval_start': '2021-06-26T15:30:00+01:00', 'interval_end': '2021-06-26T16:00:00+01:00'},
{'consumption': 0.125, 'interval_start': '2021-06-26T15:00:00+01:00', 'interval_end': '2021-06-26T15:30:00+01:00'},
{'consumption': 0.201, 'interval_start': '2021-06-26T14:30:00+01:00', 'interval_end': '2021-06-26T15:00:00+01:00'},
{'consumption': 0.043, 'interval_start': '2021-06-26T14:00:00+01:00', 'interval_end': '2021-06-26T14:30:00+01:00'},
] }

What I would like to do is loop though the data above and creating a dictionary data structure, an example of what Im trying to create is:

{
  "2021": {
    "06": {
      "01": [
        {
          "interval_start": "23:00",
          "interval_end": "23:30",
          "consumption": "0.021"
        },
        {
          "interval_start": "22:30",
          "interval_end": "23:00",
          "consumption": "0.021"
        }
      ],
      "02": [
        {
          "interval_start": "23:00",
          "interval_end": "23:30",
          "consumption": "0.021"
        },
        {
          "interval_start": "22:30",
          "interval_end": "23:00",
          "consumption": "0.021"
        }
      ]
    }
  }
}

The code which I have written to do this is:

main_obj = {}
for i in res['results']:    
    
    date = i["interval_start"].split("T")[0].split("-")

    
    insert_obj = {
        "interval_start" : i['interval_start'],
        "interval_end": i["interval_end"],
        "consumption": i["consumption"]
        
    }
    
    main_obj[date[0]] = {}
    main_obj[date[0]][date[1]] = {}
    main_obj[date[0]][date[1]][date[2]] = []
    
    main_obj[date[0]][date[1]][date[2]].append(insert_obj)
    
     
print(main_obj)

Where res['results'] is the list of Dicts above. When I print this out I am getting:

{
    '2021': {
        '06': {
            '26': [{
                'interval_start': '2021-06-26T14:00:00+01:00',
                'interval_end': '2021-06-26T14:30:00+01:00',
                'consumption': 0.043
            }]
        }
    }
}

The question I am stuck on is why when I looping through each dict is this not getting added to the list main_obj[date[0]][date[1]][date[2]]? Also, as dicts have unique keys why am I only seeing an insert for the 26 and not the 27? Which is at index 0?

Any help would be appreicated as I have been scratching my head with this for a while now!

If you are working on larger data or have multiple manipulations, you might wanna look at `pandas` library. — Vishnudev Krishnadas, Jun 27 '21 at 17:11

chepner · Answer 1 · 2021-06-27T18:04:14.743

2

You are overwriting any existing dicts/lists with the unconditional assignments like main_obj[date[0]] = {}; if date[0] has already been seen, you are erasing whatever previous data there was.

Use the setdefault method instead. (I'm not sure what the PEP-8-approved line-splitting for this looks like.)

(main_obj
  .setdefault(date[0], {})
  .setdefault(date[1], {})
  .setdefault(date[2], [])
).append(insert_obj)

edited Jun 27 '21 at 18:04

answered Jun 27 '21 at 17:12

chepner

497,756
71
530
681

There was one missing `)`. I *really* dislike explicit line continuations, and PEP-8 explicitly calls out the use of implicit line breaking inside parentheses over backslashes. – chepner Jun 27 '21 at 17:58

Alex Waygood · Answer 2 · 2021-08-21T14:06:31.427

As @chepner said, the issue here is that in every iteration of the loop you're overriding the existing value associated with a certain dictionary key, if there is an existing value associated with that key.

Here's a funky solution to this, using functools.partial and collections.defaultdict instead of the setdefault method of a regular dictionary.

from collections import defaultdict
from functools import partial
from pprint import pprint

results_list = [
    {'consumption': 0.025, 'interval_start': '2021-06-27T00:00:00+01:00', 'interval_end': '2021-06-27T00:30:00+01:00'},
    {'consumption': 0.043, 'interval_start': '2021-06-26T23:30:00+01:00', 'interval_end': '2021-06-27T00:00:00+01:00'},
    {'consumption': 0.053, 'interval_start': '2021-06-26T23:00:00+01:00', 'interval_end': '2021-06-26T23:30:00+01:00'},
    {'consumption': 0.056, 'interval_start': '2021-06-26T22:30:00+01:00', 'interval_end': '2021-06-26T23:00:00+01:00'},
    {'consumption': 0.031, 'interval_start': '2021-06-26T22:00:00+01:00', 'interval_end': '2021-06-26T22:30:00+01:00'},
    {'consumption': 0.129, 'interval_start': '2021-06-26T21:30:00+01:00', 'interval_end': '2021-06-26T22:00:00+01:00'},
    {'consumption': 0.19, 'interval_start': '2021-06-26T21:00:00+01:00', 'interval_end': '2021-06-26T21:30:00+01:00'},
    {'consumption': 0.164, 'interval_start': '2021-06-26T20:30:00+01:00', 'interval_end': '2021-06-26T21:00:00+01:00'},
    {'consumption': 0.145, 'interval_start': '2021-06-26T20:00:00+01:00', 'interval_end': '2021-06-26T20:30:00+01:00'},
    {'consumption': 0.213, 'interval_start': '2021-06-26T19:30:00+01:00', 'interval_end': '2021-06-26T20:00:00+01:00'},
    {'consumption': 0.167, 'interval_start': '2021-06-26T19:00:00+01:00', 'interval_end': '2021-06-26T19:30:00+01:00'},
    {'consumption': 0.333, 'interval_start': '2021-06-26T18:30:00+01:00', 'interval_end': '2021-06-26T19:00:00+01:00'},
    {'consumption': 0.133, 'interval_start': '2021-06-26T18:00:00+01:00', 'interval_end': '2021-06-26T18:30:00+01:00'},
    {'consumption': 0.211, 'interval_start': '2021-06-26T17:30:00+01:00', 'interval_end': '2021-06-26T18:00:00+01:00'},
    {'consumption': 0.135, 'interval_start': '2021-06-26T17:00:00+01:00', 'interval_end': '2021-06-26T17:30:00+01:00'},
    {'consumption': 0.158, 'interval_start': '2021-06-26T16:30:00+01:00', 'interval_end': '2021-06-26T17:00:00+01:00'},
    {'consumption': 0.073, 'interval_start': '2021-06-26T16:00:00+01:00', 'interval_end': '2021-06-26T16:30:00+01:00'},
    {'consumption': 0.077, 'interval_start': '2021-06-26T15:30:00+01:00', 'interval_end': '2021-06-26T16:00:00+01:00'},
    {'consumption': 0.125, 'interval_start': '2021-06-26T15:00:00+01:00', 'interval_end': '2021-06-26T15:30:00+01:00'},
    {'consumption': 0.201, 'interval_start': '2021-06-26T14:30:00+01:00', 'interval_end': '2021-06-26T15:00:00+01:00'},
    {'consumption': 0.043, 'interval_start': '2021-06-26T14:00:00+01:00', 'interval_end': '2021-06-26T14:30:00+01:00'},
]

main_obj = defaultdict(partial(defaultdict, partial(defaultdict, list)))

for i in results_list:    
    
    date = i["interval_start"].split("T")[0].split("-")    
    
    insert_obj = {
        "interval_start" : i['interval_start'],
        "interval_end": i["interval_end"],
        "consumption": i["consumption"]
        
    }

    main_obj[date[0]][date[1]][date[2]].append(insert_obj)

pprint(main_obj)  # Expected result (I think!)

The documentation for defaultdict is here, and you can read more about how it works here. When instantiating a defaultdict, the function passed to create the default has to take 0 arguments, so it's necessary to use functools.partial to modify the functions here so that they take fewer arguments than they would usually. The documentation for functools.partial is here, and you can read more about how it works here: How does functools partial do what it does?.

Looping though a list trying to create an dict

2 Answers2