2

I'm trying to get the Cartesian product of multiple arrays but the arrays are pretty large and I am trying to optimize memory usage. I have tried implementing a generator by using the below code but it just returns that there is a generator at a certain location. 

import itertools

x = [[1,2],[3,4]]

def iter_tools(*array):
    yield list(itertools.product(*array))

print(iter_tools(*x))

When I try the same code but with return instead of yield it works fine. How could I get the cartesian product by implementing a generator?

wjandrea
  • 28,235
  • 9
  • 60
  • 81
Ross Leavitt
  • 119
  • 4
  • 13
  • 1
    Your *generator materializes the whole product immediately* defeating the entire purpose. In **any** case, `itertools.product(*array)` **is already an efficient iterator**. There is no need to wrap it in an generator. So just use `itertools.product(*array)`. Your generator does work, **generator functions return generators**. It seems like you have a fundamental misunderstanding. – juanpa.arrivillaga May 08 '20 at 18:01
  • @juanpa Thanks! I posted [an answer](https://stackoverflow.com/a/61686211/4518341) partly based on your comment. – wjandrea May 08 '20 at 19:00

3 Answers3

5

Bottom line, itertools.product is already an iterator. You don't need to write your own. (A generator is a kind of iterator.) For example:

>>> x = [[1, 2], [3, 4]]
>>> p = itertools.product(*x)
>>> next(p)
(1, 3)
>>> next(p)
(1, 4)

Now, to explain, it seems like you're misunderstanding something fundamental. A generator function returns a generator iterator. That's what you're seeing from the print:

>>> iter_tools(*x)
<generator object iter_tools at 0x7f05d9bc3660>

Use list() to cast an iterator to a list.

>>> list(iter_tools(*x))
[[(1, 3), (1, 4), (2, 3), (2, 4)]]

Note how it's a nested list. That's because your iter_tools yields one list then nothing else. On that note, that part makes no sense because casting itertools.product to a list defeats the whole purpose of an iterator - lazy evaluation. If you actually wanted to yield the values from an iterator, you would use yield from:

def iter_tools(*array):
    yield from itertools.product(*array)

In this case iter_tools is pointless, but if your actual iter_tools is more complex, this might be what you actually want.

See also:

This answer is partly based on juanpa.arrivillaga's comment

wjandrea
  • 28,235
  • 9
  • 60
  • 81
1

The idea of a generator is that you don't do all the calculation at the same time, as you do with your call list(itertools.product(*array)). So what you want to do is generate the results one by one. For example like this:

def iter_tools(*array):
    for i in array[0]:
        for j in array[1]:
            yield (i, j)

You can then do something with each resulting tuple like this:

for tup in iter_tools(*x):
    print(tup)

Of course you can easily adapt the generator so that it yields each row or columns per call.

Or if you are happy with what itertools provides:

for i in itertools.product(*x):
    print(i)

What you need depends on your use-case. Hope I could help you :)

cybot
  • 101
  • 4
0

If you want to yield individual item from the cartesian product, you need to iterate over the product:

import itertools

x = [[1,2],[3,4]]

def iter_tools(*array):
    for a in itertools.product(*array):
        yield a

for a in iter_tools(*x):
    print(a)
9mat
  • 1,194
  • 9
  • 13