Python multiprocessing and progress bar function occasionally hangs

Question

I have this general function that I use to run many different operations that works to launch multiple processes and then keep track of the progress using the progressbar2 modules. Occasionally the progress bar results in a hang up and the function gets stuck. I can't seem to figure out why, if I remove the progress bar update, then the process finishes and moves on, so the problem seems to be in the while loop. Is there a better way to construct the while loop?

def runMultiProgress(function, inputList, cpus):
    from progressbar import ProgressBar, Percentage, ETA
    from time import sleep
    #setup pool
    p = multiprocessing.Pool(cpus)
    #setup progress bar
    widgets = ['     Progress: ', Percentage(),' || ', ETA()]
    pbar = ProgressBar(widgets=widgets, term_width=30, maxval=len(inputList)).start()
    #setup results and split over cpus
    results = []
    r = [p.apply_async(function, (x,), callback=results.append) for x in inputList]
    #refresh pbar every 5 seconds
    while len(results) != len(inputList):
        pbar.update(len(results))
        sleep(5)
    pbar.finish()
    p.close()
    p.join()

UPDATE: citing my sources, this multiprocessing + progress bar is from @julien-tourille answer here: Show the progress of a Python multiprocessing pool map call?

score 0 · Answer 1 · answered Dec 17 '16 at 23:51

I ended up modifying the function to use just a simple progressbar and modified how the multiprocessing was being called. This seems to be simpler and works for me.

def runMultiProgress(function, inputList, cpus):
    #setup pool
    p = multiprocessing.Pool(cpus)
    #setup results and split over cpus
    tasks = len(inputList)
    results = []
    for i in inputList:
        results.append(p.apply_async(function, [i]))
    #refresh progress every sec
    while True:
        incomplete_count = sum(1 for x in results if not x.ready())
        if incomplete_count == 0:
            break
        sys.stdout.write("Progress: %.2f%% \r" % (float(tasks - incomplete_count) / tasks * 100))
        sys.stdout.flush()
        time.sleep(1)
    p.close()
    p.join()

Python multiprocessing and progress bar function occasionally hangs

1 Answers1