Avoiding nested for loops

Question

I am trying to test some other Python code repeatedly, using all possible combinations of values for six different parameters. For each parameter I want to iterate over a range of values with a given minimum, maximum and step.

I managed to write some code like:

for var1 in range(min1, max1, step1):
    for var2 in range(min2, max2, step2):
        for var3 in range(min3, max3, step3):
            for var4 in range(min4, max4, step4):
                for var5 in range(min5, max5, step5):
                    for var6 in range(min6, max6, step6):
                        do_something_with(var1, var2, var3, var4, var5, var6)

But I do not like that the code is so deeply nested.

How can I avoid nesting multiple loops? Can it be done using recursion? How about itertools.product? I wasn't able to figure out either approach.

_{See also, more generally: Get the cartesian product of a series of lists?}

_{This question, and some answers, originally showed code for Python 2.x. It has been edited because the fundamental problem persists in 3.x, with the same solution, but xrange no longer exists - range is the drop-in replacement. For more information, see What is the difference between range and xrange functions in Python 2.X?.}

what should be actually done inside all those loops? i think it is the most important question - there are some syntax sugars to hide the loops (but still performing them), but to avoid nested loops, the most important thing is to understant the original problem — Aprillion, Jun 24 '12 at 03:34
@deathApril is right: I've given you a way to avoid the loops below, but you still have to do all those iterations. Better would be to use a different algorithm to avoid the iterations. — Ned Batchelder, Jun 24 '12 at 03:36
The point is not about using or not using for loops; it is more that you may not understand how quickly the number of test cases increases. 6 settings, each with 10 possible values, is a million test cases. — Hugh Bothwell, Jun 24 '12 at 03:49
i see the itertools.product solved your problem. good for you. just don't be surprised when you find out the performance difference between explicit nested for loops and hidden C code that performs nested loops can only be so big ;) P.S.: actually, it would be nice if you did post the performance test results :)) — Aprillion, Jun 24 '12 at 03:55
@deathApril: you're the only one talking about performance. The OP didn't mention is at all, this isn't about performance, it's about the structure of the code. — Ned Batchelder, Jun 24 '12 at 12:13

score 55 · Accepted Answer · edited Sep 29 '22 at 01:22

55

Here's how to use product:

x1 = range(min1, max1, step1)
x2 = range(min2, max2, step2)
x3 = range(min3, max3, step3)
...

for v1, v2, v3, v4, v5, v6 in itertools.product(x1, x2, x3, x4, x5, x6):
    do_something_with(v1, v2, v3, v4, v5, v6)

or a bit more compactly:

ranges = [
    range(min1, max1, step1),
    range(min2, max2, step2),
    range(min3, max3, step3),
    ...
]

for v1, v2, v3, v4, v5, v6 in itertools.product(*ranges):
    do_something_with(v1, v2, v3, v4, v5, v6)

edited Sep 29 '22 at 01:22

Karl Knechtel

62,466
11
102
153

answered Jun 24 '12 at 03:33

Ned Batchelder

364,293
75
561
662

Nice! Was about to ask a question on how to do this! – madbitloman Mar 22 '15 at 19:45
1

you could simplify this with replacing `v1, v2, v3...` with `args` then do `icky_thing(*args)` – 255.tar.xz Jun 15 '20 at 16:44
For python3 users, you should use `range` instead of `xrange`. – Brian Sep 12 '21 at 02:52

score 17 · Answer 2 · edited Sep 29 '22 at 01:23

17

You can probably use itertools.product. Something like

for var1, var2 in itertools.product(range(min1, max1, step1), range(min2, max2, step2)):
    # stuff

but with all six vars in there.

edited Sep 29 '22 at 01:23

Karl Knechtel

62,466
11
102
153

answered Jun 24 '12 at 03:31

BrenBarn

242,874
37
412
384

score -1 · Answer 3 · edited Apr 18 '22 at 01:14

you can use multiprocessing and itertools.product to speed up and simplify your code

import itertools
from multiprocessing import Pool

ranges = [
    range(min1,max1,step1),
    range(min2,max2,step2),
    range(min3,max3,step3),
    ...
]

with Pool(os.cpu_count()-2) as p:
    result = p.map(your_op_func, itertools.product(*ranges))

multiprocessing can speed up your task (because it's mutex task)
itertools.product can help you get combinations through a more effectively way

Avoiding nested for loops

3 Answers3

Linked

Related