Looking for the likely better way to get at nested data with glom?

Question

I have a particularly nasty stats object from a system that I need to retrieve data from (two of the many stats entries shown for brevity).

 'https://localhost/mgmt/tm/sys/performance/all-stats/TMM%20Memory%20Used': {'nestedStats': {'entries': {'Average': {'description': '5'},
                                                                                                         'Current': {'description': '5'},
                                                                                                         'Max(since 2019_11_12T02:47:10Z)': {'description': '5'},
                                                                                                         'Memory Used': {'description': 'TMM '
                                                                                                                                        'Memory '
                                                                                                                                        'Used'}}}},
 'https://localhost/mgmt/tm/sys/performance/all-stats/Utilization': {'nestedStats': {'entries': {'Average': {'description': '9'},
                                                                                                 'Current': {'description': '10'},
                                                                                                 'Max(since 2019_11_12T02:47:10Z)': {'description': '53'},
                                                                                                 'System CPU Usage': {'description': 'Utilization'}}}}}

Currently I use the .get method multiple times in the nested stacks, but I was listening to the author of the glom module on Talk Python this weekend and thought that might be a far cleaner solution for me. And it is, as this code makes it so that I have all the data in a loop without crazy layers of get methods (first example pictured above that I'm working on tonight). The outer key is the long URL, the inner key is the avg/current/max/desc.

stats = b.tm.sys.performances.all_stats.load()
for k, v in stats.entries.items():
    print('\n')
    spec = f'entries.{k}.nestedStats.entries'
    v_stats = glom(stats, spec)
    for k, v, in v_stats.items():
        spec = f'{k}.description'
        stat_vals = glom(v_stats, spec)
        print(f'{k}: {stat_vals}')

Which results the data I need:

Average: 5
Current: 5
Max(since 2019_11_12T02:47:10Z): 5
Memory Used: TMM Memory Used

That said, I don't really have control of the data at this point, I'm just printing it. I don't think I'm grokking the power of glom just yet and was curious if someone could point me to an example that'll help my understanding? End goal is to flatten all this data into a single list of 4 item dictionaries.

576i · Accepted Answer · 2020-02-25T18:08:48.260

First, before you try this, make sure glom is updated to the current version 19.11.0 or better.

What you ask for, is called Data-Driven-Assignment by glom's docs and not a strength of glom.

See the glom docs here

To get it to work, you may need lambdas and/or regular Python code.

Below is my working attempt, copy your example lines into the variable d.

from glom import glom, Call, T, Iter

d = { ... }  # put your example lines into this dictionary.

def get_desc(subdict):
    return {k: v.get('description', None) 
            for k,v in subdict[1]['nestedStats']['entries'].items()}

spec = (Call(list, args=(T.items(),) ), Iter().map(get_desc).all())

result = glom(d, spec)

print(result)

results in

[
{'Average': '5', 'Current': '5', 'Max(since 2019_11_12T02:47:10Z)': '5', 'Memory Used': 'TMM Memory Used'}, 
{'Average': '9', 'Current': '10', 'Max(since 2019_11_12T02:47:10Z)': '53', 'System CPU Usage': 'Utilization'}
]

UPDATE

The version below gets the same result, but avoids the need for a helper function.

What the spec does:

Call turns outer dict into a list of tuples
Iter loops over the list. For each item:
1. Take second element of tuple
2. get nestedStats.entries (which is another dict)
3. Call turns this dict into a list of tuples
4. Turn this list into a list of dicts with key and description
5. merge the list of dicts into one dict
take all the results from the iteration

I recommend trying this and removing parts of the spec to see what happens...

from glom import glom, Call, T, Iter, merge

# d = { ... }  # put your example lines into this dictionary.

spec = (
    Call(list, args=(T.items(),)),
    Iter(
        (
            T[1],
            "nestedStats.entries",
            Call(list, args=(T.items(),)),
            [{T[0]: (T[1], "description")}],
            merge,
        )
    ).all(),
)

result = glom(d, spec)

print(result)

that's helpful, thank you! I will digest that for a spell, as I'm quite confident I could walk away for a few and never arrive back at this solution. You mention that this is not a strength of glom, is there another package you can recommend that would be better suited to achieve this? — Jason Rahm, Feb 19 '20 at 21:34
@JasonRahm sorry, can't recommend anything better. I'm personally really exited about glom (and am spending a lot of time re-reading the docs over and over again), so my recommendation is to try again and again until it clicks. What still makes my brain hurt is the usage of the `T` Object, but you can substitute that for lambda and regular python functions, so glom is useful even without it. — 576i, Feb 20 '20 at 14:50
@JasonRahm updated with a full example that no longer needs the helper / lambda function. — 576i, Feb 25 '20 at 17:16

Looking for the likely better way to get at nested data with glom?

1 Answers1

UPDATE