os.walk multiple directories at once

Question

Possible Duplicate:
How to join two generators in Python?

Is there a way in python to use os.walk to traverse multiple directories at once?

my_paths = []
path1 = '/path/to/directory/one/'
path2 = '/path/to/directory/two/'
for path, dirs, files in os.walk(path1, path2):
    my_paths.append(dirs)

The above example doesn't work (as os.walk only accepts one directory), but I was hoping for a more elegant solution rather than calling os.walk twice (plus then I can sort it all at once). Thanks.

And quite appropriate, as we now have three identical answers as well as it being an identical question. — agf, Sep 28 '11 at 19:47
Exact duplicate question with exact duplicate answers! Nice! — David Heffernan, Sep 28 '11 at 19:47

score 32 · Accepted Answer · answered Sep 28 '11 at 19:40

32

To treat multiples iterables as one, use itertools.chain:

from itertools import chain

paths = ('/path/to/directory/one/', '/path/to/directory/two/', 'etc.', 'etc.')
for path, dirs, files in chain.from_iterable(os.walk(path) for path in paths):

answered Sep 28 '11 at 19:40

agf

171,228
44
289
238

Thanks, amazing! – Khalil Al Hooti Sep 20 '18 at 04:09

score 6 · Answer 2 · answered Sep 28 '11 at 19:42

6

Use itertools.chain().

for path, dirs, files in itertools.chain(os.walk(path1), os.walk(path2)):
    my_paths.append(dirs)

answered Sep 28 '11 at 19:42

David Heffernan

601,492
42
1,072
1,490

Steven Rumbalski · Answer 3 · 2011-09-28T20:57:36.803

5

Others have mentioned itertools.chain.

There's also the option of just nesting one level more:

my_paths = []
for p in ['/path/to/directory/one/', '/path/to/directory/two/']:
    for path, dirs, files in os.walk(p):
        my_paths.append(dirs)

edited Sep 28 '11 at 20:57

answered Sep 28 '11 at 19:54

Steven Rumbalski

44,786
9
89
119

I had thought about doing it that way, but I figured there was a more "pythonic" way of doing it. Thanks! – John P Sep 28 '11 at 20:55
1

I like this solution since it does not require a special lib. – wfolkerts Jun 17 '21 at 06:47
For my use case this is perfectly simple, thanks. – Justin Fortier Jul 21 '21 at 17:22
The chain solution is work but personally I like this one. Less import. – Isma Rekathakusuma Sep 22 '22 at 11:31

pyInTheSky · Answer 4 · 2011-09-28T20:25:02.087

1

since nobody mentioned it, in this or the other referenced post:

http://docs.python.org/library/multiprocessing.html

>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
...     return x*x
...
>>> p.map(f, [1,2,3])

in this case, you'd have a list of directories. the call to map would return a list of lists from each dir, you could then choose to flatten it, or keep your results clustered

def t(p):
    my_paths = []
    for path, dirs, files in os.walk(p):
        my_paths.append(dirs)


paths = ['p1','p2','etc']
p = Pool(len(paths))
dirs = p.map(t,paths)

edited Sep 28 '11 at 20:25

answered Sep 28 '11 at 20:19

pyInTheSky

1,459
1
9
24

He doesn't mean "at once" as in "at the same time" but as in "as a set" or "as a unit", so your answer doesn't really address his question. – agf Sep 28 '11 at 21:01
1

I believe it does both right? Not only do you get back your search along multiple paths as a list, which is what everyone's chain() suggestion does, but this has the added benefit of doing all these searches as a separate process. What if these are paths do unique drives. If that's the case you get even better results using this method since you are searching multiple drives simultaneously. – pyInTheSky Sep 28 '11 at 21:12

os.walk multiple directories at once

4 Answers4

Linked