0

I'm trying to use multiprocessing to run multiple scripts. At the start, I launch a loading animation, however I am unable to ever kill it. Below is an example...

Animation: foo.py

import sys
import time
import itertools

# Simple loading animation that runs infinitely.
for c in itertools.cycle(['|', '/', '-', '\\']):
    sys.stdout.write('\r' + c)
    sys.stdout.flush()
    time.sleep(0.1)

Useful script: bar.py

from time import sleep
# Stand-in for a script that does something useful.
sleep(5)

Attempt to run them both:

import multiprocessing
from multiprocessing import Process
import subprocess

pjt_dir = "/home/solebay/path/to/project" # Setup paths..
foo_path = pjt_dir + "/foo.py" # ..
bar_path = pjt_dir + "/bar.py" # ..

def run_script(path): # Simple function that..
    """Launches python scripts.""" # ..allows me to set a.. 
    subprocess.run(["python", path]) # ..script as a process.

foo_p = Process(target=run_script, args=(foo_path,)) # Define the processes..
bar_p = Process(target=run_script, args=(bar_path,)) # ..

foo_p.start() # start loading animation
bar_p.start() # start 'useful' script 

bar_p.join() # Wait for useful script to finish executing
foo_p.kill() # Kill loading animation

I get no error messages, and (my_venv) solebay@computer:~$ comes up in my terminal, but the loading animation persists (clipping over my name and environement). How can I kill it?

Solebay Sharp
  • 519
  • 7
  • 24

2 Answers2

2

I've run into a similar situation before where I couldn't terminate the program using ctrl + c. The issue is (more or less) solved by using daemonic processes/threads (see multiprocessing doc). To do this, you simply change

foo_p = Process(target=run_script, args=(foo_path,))

to

foo_p = Process(target=run_script, args=(foo_path,), daemon=True)

and similarly for other children processes that you would like to create.

With that being said, I myself am not exactly sure if this is the correct way to remedy the issue with not being able to terminate the multiprocessing program, or is it just some artifact that happens to help with this. I would suggest this thread that went into the discussion about daemon threads more. But essentially, from my understanding, daemon threads would be terminated automatically whenever their parent process is terminated, regardless of whether they are finished or not. Meanwhile, if a thread is not daemonic, then somehow you need to wait until the children processes to finish before you're able to fully terminate the program.

Remy Lau
  • 142
  • 1
  • 4
  • This now allows me to manually kill it using [CTRL] + [C] which is good news (I no longer have to constantly restart my terminal). But `foo_p.kill()` is still insufficient to end the animation on its own. This is now even more curious. Thank you for the link. – Solebay Sharp Feb 22 '22 at 20:04
1

You are creating too many processes. These two lines:

foo_p = Process(target=run_script, args=(foo_path,)) # Define the processes..
bar_p = Process(target=run_script, args=(bar_path,)) # ..

create two new processes. Let's all them "A" and "B". Each process consists of this function:

def run_script(path): # Simple function that..
    """Launches python scripts.""" # ..allows me to set a.. 
    subprocess.run(["python", path]) # ..script as a process.

which then creates another subprocess. Let's call those two processes "C" and "D". In all you have created 4 extra processes, instead of just the 2 that you need. It is actually process "C" that's producing the output on the terminal. This line:

bar_p.join()

waits for "B" to terminate, which implies that "D" has terminated. But this line:

foo_p.kill()

kills process "A" but orphans process "C". So the output to the terminal continues forever.

This is well documented - see the description of multiprocessing.terminate, which says:

"Note that descendant processes of the process will not be terminated – they will simply become orphaned."

The following program works as you intended, exiting gracefully from the second process after the first one has finished. (I renamed "foo.py" to useless.py and "bar.py" to useful.py, and made small changes so I could run it on my computer.)

import subprocess
import os

def run_script(name):
    s = os.path.join(r"c:\pyproj310\so", name)
    return subprocess.Popen(["py", s])

if __name__ == "__main__":
    useless_p = run_script("useless.py")
    useful_p = run_script("useful.py")
    
    useful_p.wait() # Wait for useful script to finish executing
    useless_p.kill() # Kill loading animation

You can't use subprocess.run() to launch the new processes since that function will block the main script until the process completes. So I used Popen instead. Also I placed the running code under an if __name__ == "__main__" which is good practice (and maybe necessary on Windows).

Paul Cornelius
  • 9,245
  • 1
  • 15
  • 24