Convert nested list comprehension to use itertools chain in python

Question

Inspired by the discussion in Making a flat list out of list of lists in Python I tried to convert this (getting files and folders modification times in some_directory):

c = [os.path.getmtime(item) for root, d, files in os.walk(some_directory)
        for item in [root] + map(lambda fi: os.path.join(root, fi), files)]

To use itertools.chain:

c = map(os.path.getmtime,
        itertools.chain.from_iterable([root] + map(lambda fi: join(root, fi), files)
                                      for root, d, files in os.walk(some_directory)))

but my profiling shows it to be slower plus does not look really elegant.

So how can I use chain in this case, that is how can I more elegantly (and faster) produce the intermediate lists ?

Or is there some other itertools function for my case ?

EDIT:

Hashed up a profiling script:

import timeit

repeat = 10

setup ="""
import itertools
import os

join = os.path.join
path = r'C:\Dropbox\eclipse_workspaces'
c = []
"""


print "Original             ", min(timeit.Timer("""[c.extend([join(root,dir) for dir in dirs] + [join(root,file) for file in files]) for root,dirs,files in os.walk(path)]""",
                       setup=setup).repeat(3, repeat))
print "For loop             ", min(timeit.Timer("""for root, d, files in os.walk(path):
    c.append(root)
    c.extend(join(root, fi) for fi in files)""",
                       setup=setup).repeat(3, repeat))
print "Comprehension        ", min(timeit.Timer('[item for r, d, f in os.walk(path) for item in [r] + map(lambda f: join(r, f), f)]',
                       setup=setup).repeat(3, repeat))
print "Comprehension + chain", min(timeit.Timer('[item for r, d, f in os.walk(path) for item in itertools.chain.from_iterable(([r], map(lambda fi: join(r, fi), f)))]',
                       setup=setup).repeat(3, repeat))
print "Itertools            ", min(timeit.Timer("""[j for j in itertools.chain.from_iterable([root] + map(lambda fi: join(root, fi), files)
                                      for root, d, files in os.walk(path))]""",
                       setup=setup).repeat(3, repeat))

seems there is no difference but I had some strange artifacts when profiling so I don't post any results. I am still interested in the fastest way that can be done, preferably using itertools

score 2 · Accepted Answer · answered Feb 27 '16 at 22:53

2

I don't think it makes sense to use chain here. chain is most useful when flattening is the whole point of the comprehension it's replacing. If you're doing more complicated stuff, then it's probably easier just to stick with a comprehension or generator expression. Or even unpack the structure into an explicit for loop, which might allow you to avoid things like the list concatenation in your current code:

c = []
for root, d, files in os.walk(some_directory):
    c.append(root)
    c.extend(join(root, fi) for fi in files)

answered Feb 27 '16 at 22:53

Blckknght

100,903
11
120
169

Thanks - I can also avoid list concatenation by some horrid thing like `itertools.chain.from_iterable(itertools.chain.from_iterable([[root], map(lambda fi: join(root, fi), files)] for root, d, files in os.walk(path)))` - I feel I miss something in itertools notation - could you rewrite my sample avoiding the `for root, d, files ....` ? – Mr_and_Mrs_D Feb 27 '16 at 23:36
Yeah, you could use an extra `chain` call to avoid the concatenation, but it will be ugly. I don't think there's any elegant way to use `chain` with the output of `os.walk`. They just don't go well together. When you want the `root` name as well as the files, it's even worse. – Blckknght Feb 27 '16 at 23:41

Convert nested list comprehension to use itertools chain in python

1 Answers1