Python itertools.product intermediate test or alternate method

Question

I am trying to use Python itertools.product to come up with all possible combinations of a long list of long lists. I do have criteria that no element of any list can be repeated. (Possible values are integers 0 to 349.)

The list to iterate contains 50 other lists.

Each of the 50 lists is shaped somewhere between (1,2) up to maxs of (57,2), (18,3), (49,4), and (5,5). These sizes will change as input data changes, but this gives you an idea of what I am dealing with to start.

Combo = list(itertools.product(*test))

remove_index = []
for i in range(0,len(Combo)): # look for duplicates in iterations, then remove
    if DupTest(Combo[i]) == True:
        remove_index.append(i)
    for j in sorted(remove_index, reverse = True):
        del Combo[j]

These are too large to be able to run itertools.product without crashing. I tried breaking it up into smaller groups (I can process 10 groups of 5 each pretty easily), but then I still can't process the next iteration with those results.

Any ideas for how to test for duplicate elements before creating the list of all the products?

Answer 1

itertools.product isn't what crashes your computer -- it's the list() around it. itertools.product returns a generator which is evaluated lazily, so it only uses memory for one item at a time. On the other hand, listing out all the combinations in the product requires enough memory for all the items in the product.

You could use filter to filter out elements of the product that don't satisfy your criteria:

for item in filter(DupTest, itertools.product(*test)):
    print(item)

---

Note some unrelated suggestions:

• Usually, you don't need to iterate for index in range(0, len(lst)) and then access item = lst[index]. Instead, get the item directly by for item in lst.
• To test if something is True, just if something: is sufficient. You don't need to specify if something == True, unless you explicitly want to exclude cases when something is a non-boolean truthy value
• As you delete elements from Combo, the old indices no longer apply, so you're not deleting the indices you originally intended. For example:

>>> lst = [0, 1, 2, 3, 4]
>>> print(lst[3])
3
>>> del lst[2]
>>> print(lst[3])
4

^{I notice you've reversed the list of indices, so this last point doesn't apply}