construct pandas multi index dataframe from nested dictionary

Question

I would like to construct pandas dataframe from nested dictionary here:

res = {'loss': {'first_neuron': {10: 0.6430850658151839,
                                 12: 0.6419735709090292,
                                 14: 0.628668905776224},
                'lr': {0.001: 0.7243950462635652,
                       0.01: 0.6431898441579607,
                       0.1: 0.5461426520789111}},
       'accuracy': {'first_neuron': {10: 0.6125457246362427,
                                     12: 0.6154635353588763,
                                     14: 0.6285751049901233},
                    'lr': {0.001: 0.5127914948963824,
                           0.01: 0.6298153875050722,
                           0.1: 0.7139774825837877}}}

to this:

i can't get it right after trying similar questions here and here

score 1 · Accepted Answer · answered Mar 25 '21 at 02:11

You dict contain many nested levels and they do not go well with pandas. You can first parse your dict into a simpler dict then manipulate the dataframe a little:

from collections import defaultdict
tmp = defaultdict(list)

for key1, value1 in res.items():
    for key2, value2 in value1.items():
        for key3, value3 in value2.items():
            tmp['metric'].append(key1)
            tmp['index1'].append(key2)
            tmp['index2'].append(key3)
            tmp['value'].append(value3)
           
df = (
    pd.DataFrame(tmp)
        .pivot(index=['index1', 'index2'], columns='metric')
        .droplevel(0, axis=1)
        .rename_axis(columns=None)
)

Result:

                     accuracy      loss
index1       index2                    
first_neuron 10.000  0.612546  0.643085
             12.000  0.615464  0.641974
             14.000  0.628575  0.628669
lr           0.001   0.512791  0.724395
             0.010   0.629815  0.643190
             0.100   0.713977  0.546143

construct pandas multi index dataframe from nested dictionary

1 Answers1