Flatten list of lists within dictionary values before processing in Pandas

Question

Issue:

If I need to flatten a list of lists I use something like this list comprehension to flatten into a single list:

[item for sublist in l for item in sublist]

I have a dictionary where some of the values are list of lists and I need to flatten these into single lists prior to importing into Pandas.

Current data:

defaultdict(list,
            {'object network fake-1': [' host 10.0.0.1'],
             'object network fake12': [' host 10.0.0.12'],
             'object network fake2': [' host 10.0.0.2 '],
             'object network fake3': [' host 10.0.0.0 255.255.255.0'],
             'object network fake4': [' host 10.0.0.4'],
             'object network fake5': [' host 10.0.0.5'],
             'object-group network prt-apps': [' network-object object fake-1',
              ' network-object object fake2',
              ' network-object object fake3',
              ' network-object object fake121'],
             'object-group network prt-apps2': [' network-object object fake4',
              ' group-object prt-apps',
              [' network-object object fake-1',
               ' network-object object fake2',
               ' network-object object fake3',
               ' network-object object fake121']],
             'object-group network prt-apps3': [' network-object object fake5',
              ' group-object prt-apps2',
              [' network-object object fake4',
               ' group-object prt-apps',
               [' network-object object fake-1',
                ' network-object object fake2',
                ' network-object object fake3',
                ' network-object object fake121']]]})

Desired data structure:

defaultdict(list,
            {'object network fake-1': [' host 10.0.0.1'],
             'object network fake12': [' host 10.0.0.12'],
             'object network fake2': [' host 10.0.0.2 '],
             'object network fake3': [' host 10.0.0.0 255.255.255.0'],
             'object network fake4': [' host 10.0.0.4'],
             'object network fake5': [' host 10.0.0.5'],
             'object-group network prt-apps': [' network-object object fake-1',
              ' network-object object fake2',
              ' network-object object fake3',
              ' network-object object fake121'],
             'object-group network prt-apps2': [' network-object object fake4',
              ' group-object prt-apps',
               ' network-object object fake-1',
               ' network-object object fake2',
               ' network-object object fake3',
               ' network-object object fake121'],
             'object-group network prt-apps3': [' network-object object fake5',
              ' group-object prt-apps2',
               ' network-object object fake4',
               ' group-object prt-apps',
                ' network-object object fake-1',
                ' network-object object fake2',
                ' network-object object fake3',
                ' network-object object fake121']})

I have searched SO for this and do not see an example that I could use. Is there a simple way to flatten these kind of 'list of list' containers within a dictionary value?

This is the way I have processed other dictionary structures when consuming in Pandas but it does not work with the first dictionary above:

pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in asa.iteritems() ]))

Possible duplicate of [Flatten (an irregular) list of lists in Python](http://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists-in-python) — Natecat, Jan 21 '17 at 23:03
Imagine you had a dictionary where all the values were integers, and you wanted to add one to all of them. How would you do this? — Natecat, Jan 21 '17 at 23:08
Updated the post showing how I used the `flatten` function in this scenario. — nipy, Jan 22 '17 at 08:09

score 2 · Answer 1 · answered Jan 22 '17 at 00:02

The following does the job as I understand it (for your specific example this relies on the list + behaviour):

def unpack(l):
    j = []
    for i in l:
        if type(i) != list:
            j.append(i)
        else:
            j = j + unpack(i)
    return j

j = {}
for k, v in l.items():
    j[k] = unpack(v)

Starting with the object as dict in your example:

l = {'object network fake-1': [' host 10.0.0.1'],
     'object network fake12': [' host 10.0.0.12'],
     'object network fake2': [' host 10.0.0.2 '],
     'object network fake3': [' host 10.0.0.0 255.255.255.0'],
     'object network fake4': [' host 10.0.0.4'],
     'object network fake5': [' host 10.0.0.5'],
     'object-group network prt-apps': [' network-object object fake-1',
                                       ' network-object object fake2',
                                       ' network-object object fake3',
                                       ' network-object object fake121'],
     'object-group network prt-apps2': [' network-object object fake4',
                                        ' group-object prt-apps',
                                        [' network-object object fake-1',
                                         ' network-object object fake2',
                                         ' network-object object fake3',
                                         ' network-object object fake121']],
     'object-group network prt-apps3': [' network-object object fake5',
                                        ' group-object prt-apps2',
                                        [' network-object object fake4',
                                         ' group-object prt-apps',
                                         [' network-object object fake-1',
                                          ' network-object object fake2',
                                          ' network-object object fake3',
                                          ' network-object object fake121']]]}

you end up with

j = {'object network fake12': [' host 10.0.0.12'],
     'object-group network prt-apps': [' network-object object fake-1',
                                       ' network-object object fake2',
                                       ' network-object object fake3',
                                       ' network-object object fake121'],
     'object network fake-1': [' host 10.0.0.1'],
     'object network fake2': [' host 10.0.0.2 '],
     'object network fake3': [' host 10.0.0.0 255.255.255.0'],
     'object-group network prt-apps2': [' network-object object fake4',
                                        ' group-object prt-apps',
                                        ' network-object object fake-1',
                                        ' network-object object fake2',
                                        ' network-object object fake3',
                                        ' network-object object fake121'],
     'object-group network prt-apps3': [' network-object object fake5',
                                        ' group-object prt-apps2',
                                        ' network-object object fake4',
                                        ' group-object prt-apps',
                                        ' network-object object fake-1',
                                        ' network-object object fake2',
                                        ' network-object object fake3',
                                        ' network-object object fake121'],
     'object network fake4': [' host 10.0.0.4'],
     'object network fake5': [' host 10.0.0.5']}

score 0 · Accepted Answer · edited May 23 '17 at 10:29

As a follow up to the original post. I managed to resolve the issue, and flattened the lists within the dictionary, with the help of the following generator function:

Taken from here:

def flatten(l):
    for el in l:
        if isinstance(el, collections.Iterable) and not isinstance(el, basestring):
            for sub in flatten(el):
                yield sub
        else:
            yield el

And using it on the dictionary as follows gave the desired output:

asa = {k: list(flatten(v)) for k, v in asa.items()}

Please note there is another version of this function for Python 3 that can be found with the link above.

Flatten list of lists within dictionary values before processing in Pandas

2 Answers2

Linked