How do I flatten nested lists of lists in python?

Question

I have created a function to split paths into lists of directories in python like so:

splitAllPaths = lambda path: flatten([[splitAllPaths(start), end] if start else end for (start, end) in [os.path.split(path)]])

with this helper function:

#these only work one directory deep
def flatten(list_of_lists):
    return list(itertools.chain.from_iterable(list_of_lists))

The output from this function looks like so:

> splitAllPaths('./dirname/dirname2/foo.bar')
[[[['.'], 'dirname'], 'dirname2'], 'foo.bar']

now I want this as a flat list. my attempts are as follows (with the output):

> flatten(splitAllPaths('./diname/dirname2/foo.bar'))
['.', 'd', 'i', 'r', 'n', 'a', 'm', 'e', 'd', 'i', 'r', 'n', 'a', 'm', 'e', '2', 'f', 'o', 'o', '.', 'b', 'a', 'r']

and

> reduce(list.__add__, (list(mi) for mi in splitAllPaths('./dirname/dirname2/foo.bar')))
me2/foo.bar')))
[[['.'], 'dirname'], 'dirname2', 'f', 'o', 'o', '.', 'b', 'a', 'r']

How do I unfold this list correctly (I would also welcome any suggestions for how to improve my splitAllPaths function)?

Desired output is `['.', 'dirname', 'dirname2', 'foo.bar']`? — Tim, Feb 12 '15 at 11:28
But why don't you just do `"./dirname/dirname2/foo.bar".split("/")`? — L3viathan, Feb 12 '15 at 11:31
because of different seperators for directories? Import `os`, and split on `os.pathsep` — L3viathan, Feb 12 '15 at 11:33
And as an exercise, it feels like i want a foldr but i don't know know to do that in python, i feel i should be ablr to work it out. — Mike H-R, Feb 12 '15 at 11:33
Foldr works on lists, though, iirc, not on weirdly nested lists. Anyways; this is definitely a terrible idea, but it works: `eval("[" + str(x).replace("[","").replace("]","") + "]")` — L3viathan, Feb 12 '15 at 11:38
@L3viathan, you're right (about splitting on pathsep), that solves the problem in a way that is platform independent (enough). it'll break on `C:\foo\bar` but it works well enough for this. I'd still like to figure out what could turn the nested lists into a normal one. — Mike H-R, Feb 12 '15 at 11:40
@L3viathan, hahaha, yes, that's a solution to the problem, I'd shudder to see anyone use it though :) — Mike H-R, Feb 12 '15 at 11:41
Anyways, take a look at [this](http://stackoverflow.com/a/16176969/1016216) — L3viathan, Feb 12 '15 at 11:44
@Leviathan, I think that last solution seems perfect. Thanks. If you want to write that (and the general os.pathsep) into an answer I'll mark it as accepted. — Mike H-R, Feb 12 '15 at 11:48

Rafael Lerm · Accepted Answer · 2015-02-12T13:12:06.720

This a less general answer, but it solves your original problem -- although its elegance is debatable.

The main idea is the fact that generating a list with the reversed (as in ['file', 'user', 'home', '/'] order is quite easy, so you can just create that and reverse it in the end. So it boils down to:

def split_paths(path):                                                       
    def split_paths_reverse(path):                                           
        head, tail = os.path.split(path)                                     
        while head and tail:                                                 
            yield tail                                                                                
            head, tail = os.path.split(head)                                 
        yield head                                                           

    return reversed(tuple(split_paths_reverse(path)))

Example:

test = '/home/user/file.txt'
print(list(split_paths(test)))

['/', 'home', 'user', 'file.txt']

You could also avoid the explicit reversing part by putting each element in a stack and then removing them, but that's up to you.

Margus · Answer 2 · 2015-02-12T12:56:38.867

1

Sortherst way that comes in mind would be:

listoflists = [[[['.'], 'dirname'], 'dirname2'], 'foo.bar']
str(listoflists).translate(None,"[]'").split(',')

edited Feb 12 '15 at 12:56

answered Feb 12 '15 at 12:38

Margus

19,694
14
55
103

It should be `, ` (note comma-space) for that example, and while it's easy (did you mean to say easiest instead of sortherst?) I'm not a fan of converting data-structures into strings (might as well be `eval`-ing them) and also this would break with path-names that contain any of the characters `[]',` – Mike H-R Feb 12 '15 at 13:07
I would be more concerned about directory names containing ',', but that that. If it does not work for you it is ok, I just aimed to demonstrate a trivial way to do this (for others it might be useful). – Margus Feb 12 '15 at 14:05
No, its good, taught me about the translate method which i wasnt aware of. Thanks – Mike H-R Feb 12 '15 at 15:06

Mike H-R · Answer 3 · 2015-02-12T12:19:56.640

I solved this by writing a (non-general) foldr. I think better, more practical solutions are provided by @L3viathan in the comments.

attempt = lambda list: attempt(list[0] + list[1:]) if len(list[0]) > 1 else list[0] + list[1:]

Output

> attempt([[[['.'], 'dirname'], 'dirname2'], 'foo.bar'])
['.', 'dirname', 'dirname2', 'foo.bar']

I've also now written it in terms of a general foldr1

> foldr1 = lambda func, list: foldr1(func, func(list[0], list[1:])) if len(list[0]) > 1 else func(list[0], list[1:])
> foldr1(list.__add__, [[[['.'], 'dirname'], 'dirname2'], 'foo.bar'])
['.', 'dirname', 'dirname2', 'foo.bar']

NOTE: Could someone more familiar than me confirm that this is a foldr and not a foldl (I often get them confused).

How do I flatten nested lists of lists in python?

3 Answers3