0

i see a lot of examples of how to use multiprocessing but they all talk about spawning workers and controlling them while the main process is alive. my question is how to control background workers in the following way:

start 5 worker from command line:

  • manager.py --start 5

after that, i will be able to list and stop workers on demand from command line:

  • manager.py --start 1 #will add 1 more worker
  • manager.py --list
  • manager.py --stop 2
  • manager.py --sendmessagetoall "hello"
  • manager.py --stopall

the important point is that manager.py should exit after every run. what i don't understand is how to get a list of already running workers from an newly created manager.py program and communicate with them.

edit: Bilkokuya suggested that i will have (1)a manager process that manage a list of workers... and will also listen to incoming commands. and (2) a small command line tool that will send messages to the first manager process... actually it sounds like a good solution. but still, the question remains the same - how do i communicate with another process on a newly created command line program (process 2)? all the examples i see (of Queue for example) works only when both processes are running all the time

ponip
  • 1
  • 1
  • Relevant: [how-to-handle-the-signal-in-python-on-windows-machine](https://stackoverflow.com/questions/35772001/how-to-handle-the-signal-in-python-on-windows-machine) – stovfl Sep 12 '17 at 14:15

1 Answers1

0

The most portable solution I can suggest (although this will still involve further research for you), is to have a long-running process that manages the "background worker" processes. This shouldn't ever be killed off, as it handles the logic for piping messages to each sub process.

Manager.py can then implement logic to create communication to that long-running process (whether that's via pipes, sockets, HTTP or any other method you like). So manager.py effectively just passes on a message to the 'server' process "hey please stop all the child processes" or "please send a message to process 10" etc.

There is a lot of work involved in this, and a lot to research. But the main thing you'll want to look up is how to handle IPC (Inter-Process Communication). This will allow your Manager.py script to interact with an existing/long-running process that can better manage each background worker.

The alternative is to rely fully on your operating system's process management APIs. But I'd suggest from experience that this is a much more error prone and troublesome solution.

  • So you suggest that i will have (1)a manager process that manage a list of workers... and will also listen to incoming commands. and (2) a small command line tool that will send messages to the first manager process... actually it sounds like a good solution. but still, the question remains the same - how do i communicate with another process on a newly created command line program (process 2)? all the examples i see (of Queue for example) works only when both processes are running all the time – ponip Sep 12 '17 at 11:26
  • Correct, the manager process will be "always-on" (like a windows service - which you may wish to look into). The commandline tool will need to use some form of IPC (Inter Process Communication) to then message it while it runs. This depends a lot on your Operating System and design decisions (do you need them to run on the same computer => pipes, do you want network access => sockets/HTTP). If you look into how an HTTP server works (at a high level), perhaps it will give you a better idea of how this can be structured. –  Sep 12 '17 at 12:27
  • i use windows and actually i'm surprised that its difficult to find a simple pipes communication example for windows to support communication between 2 scripts, so i guess i will implement it with sockets. thanks – ponip Sep 12 '17 at 12:50