0

I'm trying to display multiple progress bars for parallel processes inside a jupyter notebook. The following works in a normal python console and correctly displays two parallel error bars:

funcs.py

def product_(a, b):
    for _ in tqdm(range(10)):
        sleep(0.1)
    return a * b

main.py

from time import sleep
from funcs import product_
import multiprocessing

with multiprocessing.Pool(processes=2) as pool:
    results = pool.starmap(product_, iter([(1, 2), (3, 4)]))

But the equivalent in a jupyter notebook (replacing import tqdm by import tqdm.notebook) doesn't show anything.

Note that I don't want a progress bar over processes, but over a loop inside each of the processes.

Eudald
  • 358
  • 3
  • 12
  • Look at this answer: https://stackoverflow.com/questions/7714868/multiprocessing-how-can-i-%CA%80%E1%B4%87%CA%9F%C9%AA%E1%B4%80%CA%99%CA%9F%CA%8F-redirect-stdout-from-a-child-process – Farhood ET Aug 05 '21 at 10:31
  • Thanks, but this is a bit overkill, no? I have found a bunch of fairly simple solutions (eg https://github.com/tqdm/tqdm/issues/485), but they don't seem to work for me, probably because I'm doing something wrong... – Eudald Aug 05 '21 at 10:44
  • `stuff`? Do you mean `product_`? Anyway, if you want to run a function (I am assuming it is `product_`) in parallel using multiprocessing under `jupyter-notebook`, that function must be imported from another module; this is a peculiarity of `jupyter-notebook`. Check your `jupyter-notebook` start-up window and you will see error messages. – Booboo Aug 05 '21 at 10:53
  • Yeah, sorry about the stuff. In my real file it's in another module, I just put them together for simplicity but I have now edited the question. – Eudald Aug 05 '21 at 10:57
  • I was about to issue a close vote since this appeared to be a duplicate of [Jupyter notebook never finishes processing using multiprocessing (Python 3)](https://stackoverflow.com/questions/47313732/jupyter-notebook-never-finishes-processing-using-multiprocessing-python-3). You should be careful with the shortcuts you take for "simplicity." – Booboo Aug 05 '21 at 10:59
  • By the way, when I run your "normal python console" version, the two progress bars do not display, in parallel. The first run runs and progresses every .1 seconds and when it completes only then does the second bar display already completed. You need to specify a unique *position=number* argument for each invocation of `tqdm` to have these progress in parallel. – Booboo Aug 05 '21 at 11:29
  • But even that doesn't work well. If you give each bar a different delay so that one finishes before the other, e.g. position=0 finished first, then the bar at position=1 is the only bar left running and `tqdm` stops drawing that bar and starts a new bar below that one as if once the bar at position=0 is done it no longer exists and the bar that being drawn at position=1 now appears to be at positiion=0 and thus `tqdm` has to start all over. – Booboo Aug 05 '21 at 11:58
  • Yeah, @Booboo, that's why I didn't give it a position. Running them overlapping each other is definitely better than having them start all over all the time... – Eudald Aug 05 '21 at 13:14

0 Answers0