How to create nested generator structures in python?

Question

I'm trying to create an ImageSeries object, where I want to retrieve images in a certain pattern (for every value in xy, and every value in z), and I call methods that will append a generator to a tasks list, and run the generator through two for loops to do this.

But my second task gets exhausted after the first iteration of the first task, which is not my desired outcome. I want the second task to run every iteration of the first task.

I was wondering if there are efficient ways to program patterns like this.

class ImageSeries:
    tasks = []

    def xy(self, position):
        print(position)
        yield "xy"

    def z(self, position):
        print(position)
        yield "z"

    def xy_scan(self, positions):
        self.tasks.append((self.xy(pos) for pos in positions))

    def z_scan(self, positions):
        self.tasks.append((self.z(pos) for pos in positions))

    def run(self):
        for i in self.tasks[0]:
            next(i)
            for j in self.tasks[1]:
                next(j)

    def __repr__(self):
        return str(self.tasks)
    

if __name__ == "__main__":
    s = ImageSeries()
    positions = [[0, 0], [100, 100], [1000, 1000]]
    s.xy_scan(positions)
    s.z_scan([0, 100, 1000, 10000])

Current output:

[0, 0]
0
100
1000
10000
[100, 100]
[1000, 1000]

Expected output:

>>> s.run()
[0, 0]
0
100
1000
10000
[100, 100]
0
100
1000
10000
[1000, 1000]
0
100
1000
10000

what is the current output, and what is the output you want? — Gulzar, Jul 18 '20 at 16:48

Gulzar · Accepted Answer · 2020-07-18T17:40:37.687

Here you go

class ImageSeries:
    def __init__(self):
        self._xy_tasks = None
        self._z_tasks = None

    def xy(self, position):
        print(position)
        yield "xy"

    def z(self, position):
        print(position)
        yield "z"

    def xy_scan(self, positions):
        self._xy_tasks = lambda: (self.xy(pos) for pos in positions)

    def z_scan(self, positions):
        self._z_tasks = lambda: (self.z(pos) for pos in positions)

    def run(self):
        for xy_generator in self._xy_tasks():
            next(xy_generator)
            for z_generator in self._z_tasks():
                next(z_generator)

    def __repr__(self):
        return str(self._xy_tasks()) + " " + str(self._z_tasks())


if __name__ == "__main__":
    s = ImageSeries()
    positions = [[0, 0], [100, 100], [1000, 1000]]
    s.xy_scan(positions)
    s.z_scan([0, 100, 1000, 10000])
    s.run()

Did some things:

called run()
self.tasks made no sense as a list, becuase each cell has a different meaning, so I separated it into two separate member variables.
The main thing was making sure the generator would be created again on each run, as it cannot be reset. I accomplished that by using a lambda, so you can call a function that creates the generator every time, instead of the generator itself. Notice the self._xy_tasks(). This calls a function that creates the generator.

I use `tasks` list to keep track of the order of the methods in `ImageSeries` as they were called. Sometimes I might have to do a `s.z_scan` before a `s.xy_scan`. I also have a `s.exp_scan` I want to add in future, which wouldn't work well with the way for loops are written in this. But it works otherwise! — pskeshu, Jul 18 '20 at 17:49
I guess it took some time for me to understand your answer. It's rather clever, especially 3. Thanks :) — pskeshu, Jul 18 '20 at 18:38
Is there a way I can use something like `itertools.product()` to generalize iteration for more than 2 nested for loops? — pskeshu, Jul 18 '20 at 18:43

score 0 · Answer 2 · answered Jul 18 '20 at 17:34

Generators don't know they are nested. After the first time the generator was exhausted, it is over. In fact you don't need generators in this case as they make sence when you are iterating a long list you don't want to store in memory. But here you do have to store in memory all the repeated sequences. You can use generator only at the upper level loop. But it only makes sence if it is really long and recieved from some stream. If it is already in memory, you don't really need generators. All you want can be done much simpler

xy_list = [[0, 0], [100, 100], [1000, 1000]]
z_list = [0, 100, 1000, 10000]
for xy in xy_list:
    print(xy)
    for z in z_list:
        print(z)

If you need it to be a class, just use xy_scan, z_scan to save to self.xy_list, self.z_list and use the same for loop in run method (just add self. to xy_list and z_list)

I'm using generators to interleave code in-between yield statements, so that pattern works best for me. — pskeshu, Jul 18 '20 at 17:51
You can put the code you want in the `for` loop near `print`. — Leonid Mednikov, Jul 18 '20 at 18:38

How to create nested generator structures in python?

2 Answers2