Most efficient way to convert/unpack an itertools.chain object to an unordered and ordered list

Question

Other than using the list and sorted methods to convert an itertools.chain object to get an unordered and ordered list, respectively, are there more efficient ways of doing the same in python3? I read in this answer that list is for debugging. Is this true?

Below is an example code where I time the processes:

from itertools import chain
from time import time

def foo(n):
        for i in range(n):
            yield range(n)

def check(n):
    # check list method
    start = time()
    a = list(itertools.chain.from_iterable(foo(n)))
    end = time()- start
    print('Time for list = ', end)
    # check sorted method
    start = time()
    b = sorted(itertools.chain.from_iterable(foo(n)))
    end = time()- start
    print('Time for sorted = ', end)

Results:

>>> check(1000)
Time for list =  0.04650092124938965
Time for sorted =  0.08582258224487305
>>> check(10000)
Time for list =  1.615750789642334
Time for sorted =  8.84056806564331
>>>

I don't know what he meant with "debugging purposes", but I doubt you can get any faster than that — BlackBear, Feb 12 '17 at 13:04

Mazdak · Answer 1 · 2017-02-12T13:18:46.590

The most efficient way is using list() but if you want to flatten a nested iterable by itertools.chain() or concatenate some iterables and then convert it to list you can just use a nested list comprehension at once. Also the reason that sorted() takes more time is that it's sorting the iterable while list just calls some methods of generator function (like __next__) in order to copy the items to a list object.

Note that in terms of run time the itertools.chain can perform slightly even faster than list comprehension (python 2.x and python 3.x). Here is an example:

In [27]: lst = [range(10000) for _ in range(10000)]

In [28]: %timeit [i for sub in lst for i in sub]
1 loops, best of 3: 3.94 s per loop

In [29]: %timeit list(chain.from_iterable(lst))
1 loops, best of 3: 2.75 s per loop

score 2 · Accepted Answer · answered Feb 12 '17 at 13:16

Other than using the list and sorted methods to convert an itertools.chain object to get an unordered and ordered list, respectively, are there more efficient ways of doing the same in python3?

simple answer: no. When working with the python generators and iterators, the only caveat you have to avoid is to convert a generator into a list, then into a generator, then into a list again etc…

i.e. a chain like that would be stupid:

list(sorted(list(filter(list(map(…

because you would then lose all the added value of the generators.

I read in this answer that list is for debugging. Is this true?

it depends on your context, generally speaking a list() is not for debugging, it's a different way to represent an iterable.

You might want to use a list() if you need to access an element at a given index, or if you want to know the length of the dataset. You'll want to not use a list() if you can consume the data as it goes.

Think of all the generators/iterator scheme as a way to apply an algorithm for each item as they are available, whereas you work on lists as bulk.

About the question you quote, the question is very specific, and it asks how it can introspect a generator from the REPL, in order to know what is inside. And the advice from the person who answered this is to use the list(chain) only for introspection, but keep as it was originally.

Most efficient way to convert/unpack an itertools.chain object to an unordered and ordered list

2 Answers2