30

I am using itertools to run a numerical simulation iterating over all possible combinations of my input parameters. In the example below, I have two parameters and six possible combinations:

import itertools

x = [0, 1]
y = [100, 200, 300]

myprod = itertools.product(x, y)

for p in myprod:
    print p[0], p[1]
    # run myfunction using p[0] as the value of x and p[1] as the value of y

How can I get the size of myprod (six, in the example)? I'd need to print this before the for loop starts.

I understand myprod is not a list. I can calculate len(list(myprod)), but this consumes the iterator so the for loop no longer works.

I tried:

myprod2=copy.deepcopy(myprod)
mylength = len(list(myprod2))

but this doesn't work, either. I could do:

myprod2=itertools.product(x,y)
mylength = len(list(myprod2))

but it's hardly elegant and pythonic!

jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
Pythonista anonymous
  • 8,140
  • 20
  • 70
  • 112
  • Getting the length of an iterator doesn't make sense, and kind of spoils the point of using one! You *could* work it out, though... *"this uses up (what is the proper term?)"* - *"consumes"*, generally. – jonrsharpe Aug 18 '15 at 14:00
  • possible duplicate of [Is there any built-in way to get the length of an iterable in python?](http://stackoverflow.com/questions/390852/is-there-any-built-in-way-to-get-the-length-of-an-iterable-in-python) – Frerich Raabe Aug 18 '15 at 14:01
  • well, it does make a lot of sense in my specific case because (for reasons too long to explain here) I need to get the total number of combinations BEFORE the for loop starts – Pythonista anonymous Aug 18 '15 at 14:04
  • But *in general*, as iterators aren't necessarily finite, you can't find out the length without consuming them. – jonrsharpe Aug 18 '15 at 14:05
  • Sure, but what would you recommend for my specific case, in which my iterator is always finite? Would you recommend using something other than itertools? – Pythonista anonymous Aug 18 '15 at 14:07
  • Also, why does the deep copy not work? – Pythonista anonymous Aug 18 '15 at 14:08
  • Because the `deepcopy` process still consumes the iterator, it doesn't create another one. You could use [`itertools.tee`](https://docs.python.org/3/library/itertools.html#itertools.tee) to *"copy"* an iterator but, as you're consuming it all up-front, you already lost the memory benefits straight away (see the notes on that function in the docs). – jonrsharpe Aug 18 '15 at 14:10

4 Answers4

18

To implement Kevin's answer for an arbitrary number of source iterables, combining reduce and mul:

>>> import functools, itertools, operator
>>> iters = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> functools.reduce(operator.mul, map(len, iters), 1)
27
>>> len(list(itertools.product(*iters)))
27

Note that this will not work if your source iterables are themselves iterators, rather than sequences, for the same reason your initial attempts to get the length of the itertools.product failed. Python generally and itertools specifically can work in a memory-efficient way with iterators of any length (including infinite!) so finding out lengths up-front isn't really a case it was designed to deal with.

Community
  • 1
  • 1
jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
10

While this doesn't answer the question directly, very often we want to find the length of generators to estimate the progress/runtime.

For this, do consider using tqdm's (version >= 4.42.0) wrappers around generator functions that don't forget the lengths of iterators (tqdm is a progressbar library). E.g.,

from tqdm.contrib.itertools import product
from time import sleep
for i, j in product(range(3), range(4)):
    sleep(1)

will show a progress bar. The length of the product is shown as the total of the tqdm object (e.g.., the 6 in 3/6 [00:03<00:03] shown).

enter image description here

LHeng
  • 531
  • 5
  • 9
4

How about:

mylength = len(x) * len(y)
Kevin
  • 74,910
  • 12
  • 133
  • 166
  • less convenient in my specific case, because I don't have two inputs but many many more – Pythonista anonymous Aug 18 '15 at 14:02
  • 1
    @Pythonistaanonymous but it's still just the product of the lengths of the source iterables (`functools.reduce(operator.mul, map(len, iters), 1)` in Python 3.x). – jonrsharpe Aug 18 '15 at 14:05
  • 2
    I'm down-voting this because it doesn't answer the question. For example, I have a program that passes around an itertools.product. This clever trick won't work for that. – Mikhail Aug 01 '16 at 03:18
1

Alternative solution I used:

import itertools

param = (('a', 'b'), (1, 2)) # a list of lists

# Calculate all combinations
combinations = itertools.product(*param)

# Calculate number of combinations
total_combinations = 1
for i in param:
    total_combinations = total_combinations * len(i)
Tom Saenen
  • 141
  • 7