Confused by chain enumeration

Question

I wanted to gather all of the header files in a list of subdirectories. However, if I do

from glob import glob
from itertools import chain

subDirs = ['FreeRTOS', 'Twig']
for each in chain(glob(eachDir+'/*.h') for eachDir in subDirs):
  print each

What I get is

['FreeRTOS/croutine.h', 'FreeRTOS/FreeRTOS.h', 'FreeRTOS/FreeRTOSConfig.h', 'FreeRTOS/list.h', 'FreeRTOS/mpu_wrappers.h', 'FreeRTOS/portable.h', 'FreeRTOS/portmacro.h', 'FreeRTOS/projdefs.h', 'FreeRTOS/queue.h', 'FreeRTOS/semphr.h', 'FreeRTOS/StackMacros.h', 'FreeRTOS/task.h', 'FreeRTOS/timers.h']
['Twig/twig.h']

But what I wanted to see was

'FreeRTOS/croutine.h'
'FreeRTOS/FreeRTOS.h'
'FreeRTOS/FreeRTOSConfig.h'
'FreeRTOS/list.h'
'FreeRTOS/mpu_wrappers.h'
'FreeRTOS/portable.h'
'FreeRTOS/portmacro.h'
'FreeRTOS/projdefs.h'
'FreeRTOS/queue.h'
'FreeRTOS/semphr.h'
'FreeRTOS/StackMacros.h'
'FreeRTOS/task.h'
'FreeRTOS/timers.h'
'Twig/twig.h'

I thought that was what the chain() would do for me. What am I missing?

unutbu · Accepted Answer · 2013-02-11T20:41:44.323

3

I think you are looking for itertools.chain.from_iterable:

import os
import glob
import itertools

for each in itertools.chain.from_iterable(
        glob.glob(os.path.join(eachDir,'/*.h')) 
        for eachDir in subDirs):
    print each

It flattens an iterable of iterables:

In [6]: import itertools as IT
In [7]: list(IT.chain.from_iterable([['a', 'b', 'c'], [1, 2, 3]]))
Out[7]: ['a', 'b', 'c', 1, 2, 3]

edited Feb 11 '13 at 20:41

answered Feb 11 '13 at 20:30

unutbu

842,883
184
1,785
1,677

score 0 · Answer 2 · edited May 23 '17 at 11:48

Probably slower than unutbu's answer, but this would also work:

import collections
def flattenIter(l):
    """
        Iterator that flattens a list by one each iteration
        To get a fully flattened list of L do:
            list(flattenIter(L))
    """
    for el in l:
        if isinstance(el, collections.Iterable) and not isinstance(el, basestring):
            for sub in flattenIter(el):
                yield sub
        else:
            yield el

Note that compared to unutbu's answer this flattens any amount of lists in list as opposed to his method. Example:

print list(IT.chain.from_iterable([['a', 'b', 'c'], [1, 2, 3],"aap",["beer",[1]]]))
# ['a', 'b', 'c', 1, 2, 3, 'a', 'a', 'p', 'beer', [1]]
print list(flattenIter([['a', 'b', 'c'], [1, 2, 3],1,"aap",["beer",[1]]]))
# ['a', 'b', 'c', 1, 2, 3, 1, 'aap', 'beer', 1]

Same 'out' as the unutbu's answer.

In : list(flattenIter([['a', 'b', 'c'], [1, 2, 3]]))
Out: ['a', 'b', 'c', 1, 2, 3]

Here is a speed comparison with unutbu's version:

import timeit
import collections
import time
import itertools as IT

class Timer:
    def __enter__(self):
        self.start = time.clock()
        return self

    def __exit__(self, *args):
        self.end = time.clock()
        self.interval = self.end - self.start

def flattenIter(l):
    """
        Iterator that flattens a list by one each iteration
        To get a fully flattened list of L do:
            list(flattenIter(L))
    """
    for el in l:
        if isinstance(el, collections.Iterable) and not isinstance(el, basestring):
            for sub in flattenIter(el):
                yield sub
        else:
            yield el

with Timer() as t:
    for x in xrange(10000):
        list(IT.chain.from_iterable([['a', 'b', 'c'], [1, 2, 3]]))

print t.interval
# result: 0.0220727116414

with Timer() as t:
    for x in xrange(10000):
        list(flattenIter([['a', 'b', 'c'], [1, 2, 3]]))

print t.interval
# result: 0.147218201587

Also check out:

Making a flat list out of list of lists in Python

It's marked as duplicate though it has some other solid links in there as well. ;)

Confused by chain enumeration

2 Answers2