How can I make this generator loop?

Question

I am trying to loop over a directory and load all files. I've tried using one generator to load files and another one to generate batches and call the first generator when it runs of out memory.

def file_gen(b):

    # iterate over my directory and load two audio file at a time

    for n in range(len(b)):
        path_ = os.path.join(os.path.join(path,'Mixtures'), 'Dev')
        os.chdir(os.path.join(path_,b[n]))
        y, _ = librosa.load('mixture.wav', sr=rate)

        path_vox = os.path.join(os.path.join(path,'Sources'), 'Dev')
        os.chdir(os.path.join(path_vox,b[n]))
        x, _ = librosa.load('vocals.wav', sr=rate)

        yield y, x

list_titles = os.listdir(os.path.join(os.path.join(path,'Mixtures'),'Dev'))

gen_file = file_gen(list_titles)

# second generator

def memory_test():

    memory = 0

    if memory == 0:
        a, b = next(gen_file)
        a, _ = mag_phase(spectrogram(a))
        b, _ = mag_phase(spectrogram(b))

        # calculate how many batches I can generate from the file

        memory = a.shape[1]/(n_frames*(time_len-overlap) + time_len)

    for n in range(memory):
        yield memory
        memory = memory -1

test = memory_test()

The second generator is where the problem is. Ideally, I would like both generator to iterate indefinitely though (the first one should go back to the beginning of the list).

Thank you!

Possible duplicate of [Resetting generator object in Python](https://stackoverflow.com/questions/1271320/resetting-generator-object-in-python) — MyNameIsCaleb, Apr 19 '19 at 18:44
unrelated: you can simplify your `os.path.join(..., os.path.join(...))` quite a bit: `print(os.path.join("tata", "mix","dev","uuups")) # tata/mix/dev/uuups` — Patrick Artner, Apr 19 '19 at 18:54

MyNameIsCaleb · Accepted Answer · 2019-04-21T17:29:30.867

1

itertools.cycle()
One way you could do this is to use itertools.cycle() which will essentially store the results of the generator and then continuously loops them over and over. docs

If you chose to do that, you would consume a lot of additional memory storing those results.

except StopIteration
As an alternative method, you could try: and except StopIteration for your generator yield in order to reset it back to the beginning. Generators always raise StopIteration if you call __next__ on an exhausted generator.

Edit: I originally linked to a wrapper function here but the code in that example actually doesn't work. Below is code that I have tested to work which is hopefully helpful. My answer here is based on the same concept.

def Primes(max):   # primary generator
    number = 1
    while number < max:
        number += 1
        if check_prime(number):
            yield number

primes = Primes(100)

def primer():   # this acts as a loop and resets your generator
    global primes
    try:
        ok = next(primes)
        return ok
    except StopIteration:
        primes = Primes(100)
        ok = next(primes)
        return ok

while True:   # this is the actual loop continuing forever
    primer()

You'll notice we couldn't implicitly refer to our own function in order to reset itself, and we also couldn't use a standard for loop because it will always catch StopIteration before you can, by design [more info].

edited Apr 21 '19 at 17:29

answered Apr 19 '19 at 18:49

MyNameIsCaleb

4,409
1
13
31

Thank you for the answer. I can see now that I repeated the question, which I am sorry for. However, I find it difficult to wrap my head around the second option. Using the code from the example, when I run next(generator_generating_function()) I keep getting keep getting the first item returned. Any advice on that? – Jan Gryga Apr 21 '19 at 00:34
Thank you! Helped a lot. – Jan Gryga Apr 22 '19 at 23:20
Is there a way to still use it as a generator object? I am using a function from Keras called fit_generator that takes a generator object. When I try to use the wrapper() function instead, it throws an error: ValueError: You must specify `batch_size` – Jan Gryga Apr 24 '19 at 12:36
I wrapped it in another generator, which means that I'll have to calculate the amount of batches I can generate and then multiply it by the amount of epochs, but it works, so I guess I should stick with it for now – Jan Gryga Apr 24 '19 at 13:03
Yea I'm not sure that you can stop a generator inside of itself, I tried it a few ways and couldn't get it to work, which is why I wrapped it outside with another function. And with that the only way would be to wrap it with a generator as well. Interesting problem. – MyNameIsCaleb Apr 24 '19 at 13:18

How can I make this generator loop?

1 Answers1