0

I don't understand why but every 1 out of 10 times I run this code I get a RuntimeError: Set changed size during iteration.

I essentially go through an input set. Do parallel jobs with impa_unordered, because I don't care of the order and because I want to restart a process if it's taking too long. Here is the code:

from multiprocessing import Pool
import time
import multiprocessing
import numpy as np
from random import choice

def fun(x):
    
    y=choice([.000001,.2])
    time.sleep(y)
    return x,y


if __name__=='__main__':
    TIMEOUT = .1
    results_list = []
    input_list = set(np.arange(100))
                    
    while len(input_list)!=0:
        print('New iteration')
        print(len(input_list))
        num_processors = min(max([multiprocessing.cpu_count() - 1, 2]),len(input_list))
        print('Cores: ',num_processors)



        with Pool(processes=num_processors) as pool:
            time.sleep(0.01)
            results = pool.imap_unordered(fun, input_list)  
            try:
                while True:
                    x,y = results.next(timeout=TIMEOUT)
                    results_list.append(y)
                    input_list.remove(x)

            except (StopIteration, KeyboardInterrupt):
                #pool.terminate()
                break

            except multiprocessing.TimeoutError:
                #pool.terminate()
                print("Timeout")
                continue
                

    print(genes_list,results_list)
    print('DOne')
M. GENTILI
  • 53
  • 2
  • 7
  • 1
    An initial guess would be that you are calling input_list.remove while pool.imap_unordered is still iterating over input_list. Perhaps simply removing that line removing from the input_list would be sufficient to avoid the crash? – Andrew McClement Mar 17 '21 at 15:50
  • Does this answer your question? [Strange result when removing item from a list while iterating over it](https://stackoverflow.com/questions/6260089/strange-result-when-removing-item-from-a-list-while-iterating-over-it) from the answer: *"There's no good way to alter a list's length while iterating over it."* – Aaron Mar 17 '21 at 17:18

0 Answers0