1

I'm trying to couple the execution of a spawned process in a worker pool to a new system terminal. In the following example (adapted from @sylvain-leroux's answer to this question) a pool of workers is constructed to do some work with queued objects.

import os
import time
import multiprocessing

# A main function, to be run by our workers.
def worker_main(queue):
    print('The worker at', os.getpid(), 'is initialized.')
    while True:

        # Block until something is in the queue.
        item = queue.get(True)
        print(item)
        time.sleep(0.5)

if __name__ == '__main__':

    # Instantiate a Queue for communication.
    the_queue = multiprocessing.Queue()

    # Build a Pool of workers, each running worker_main.
    the_pool = multiprocessing.Pool(3, worker_main, (the_queue,))

    # Iterate, sending data via the Queue.
    for i in range(5):
        the_queue.put("That's a nice string you got there.")
        the_queue.put("It'd be a shame if something were to... garble it.")

    worker_pool.close()
    worker_pool.join()
    time.sleep(10)

If you run this from a system terminal you'll see a bunch of garbled text, because each of the workers is writing out to, and executing in, the same console. For an project I'm working on, it would be helpful to spawn a new shell/console to host each worker process, such that all printed output is displayed in that shell, and the execution of the worker process is host in that shell. I've seen several examples doing this with Popen using the shell keyword, but I need to stick to a pool-based implementation, due to compatibility constraints. Has anyone out there done this? Guidance is appreciated.

Community
  • 1
  • 1
Justin Fletcher
  • 2,319
  • 2
  • 17
  • 32
  • on Linux you could write to separated files and manually open shells to use command `watch -n 1 cat file.txt` to display new content of files every one second. – furas Jan 01 '17 at 03:59
  • Thanks @furas. That's a useful idiom to know. I usually work on Linux, but I would like this solution to be cross-platform, for extensibility. Also, though this would give me access to the printed information, it wouldn't result in the execution being tied to the watching shell, which is desirable for my application. – Justin Fletcher Jan 01 '17 at 04:25

1 Answers1

1

Try using the Queue the other way around.

Let the workers put messages into the Queue, and in the parent process get them from the Queue and print them. That should get rid of intermingled output.

If you want to pass messages both from parent to workers and back, use two queues. One for passing messages to workers, and one to pass messages back to the parent.

Roland Smith
  • 42,427
  • 3
  • 64
  • 94
  • That's an interesting use of Queue that I hadn't considered - it will probably come in handy. Unfortunately, it won't do in this case. I still need to pass information that only the parent process has to the workers via that queue, in order to do useful work. In this example all the workers do is print, but in reality they will do something with the information. I'm trying to decouple the spawned worker processes from the terminal in which the pool was instantiated. The garbled text is a symptom of the coupling, but not the whole problem. I've amended the question to be more clear. Thanks! – Justin Fletcher Jan 03 '17 at 00:26
  • @JustinFletcher Then just use two `Queue`s. See updated answer. – Roland Smith Jan 04 '17 at 00:57