python multiprocessing starmap progressbar

Question

Since I need several args in my worker function, so I use starmap, but how can I show the progress with tqdm?

from itertools import repeat
from multiprocessing import Pool

def func(i, a, b, c):
    print(i, a, b, c)

if __name__ == '__main__':  
    pool = Pool(thread_num)
    a, b, c = 'a', 'b', 'c'
    pool.starmap(func, zip(range(100), repeat(a), repeat(b), repeat(c)))
    pool.close()
    pool.join()

So How can I user tqdm to show the pregress?

Is it possible to change your function to receive a single tuple argument rather than several? That would let you `imap` instead of `starmap`, and so the main process could loop over the incoming results to update the progress bar. Unfortunately there's no `istarmap`! If you only need to do this for one function that you can't modify (e.g. it's from a library), you could wrap it with a `def wrapper(tup): func(*tup)` function defined at the top level. — Blckknght, May 01 '17 at 07:40
[Here](https://stackoverflow.com/a/57364423/9059420) you'll find a patch adding `.istarmap()` and an example with `tqdm`. — Darkonaut, Aug 06 '19 at 05:37

score -1 · Answer 1 · answered May 01 '17 at 05:31

You should create a process to monitor the signal passed by other processes and update your tqdm. A minimal example for you :

from multiprocessing import Queue, Pool, Process

def listener(q, num):
    tbar = tdqm(total = num)
    for i in iter(q.get, None):
        tbar.update()
    tbar.close()

def worker(q):
    do something.
    queue.put(1)

if __name__ == "__main__":
    ....
    q.put(None)  #when you finish your work, put a None signal to finish the listener.

The answer should be more direct and relevant to the question and suggest the changes in the code itself. — user3480922, Dec 10 '19 at 17:26

python multiprocessing starmap progressbar

1 Answers1