0

I'm using Python 3. I have the following code that tries to catch CTRL+C when running a pool of async workers. Each worker just runs in an infinite loop waiting for messages to show up in a queue for processing. What I don't understand is why the log.info inside the except block and print message are not printed when I press CTRL+C. It just drops me back into the xterm. Is the with block doing something to prevent this?

 def worker(q):
            """Worker to retrieve item from queue and process it.

            Args:
                q -- Queue
            """

            # Run in an infinite loop. Get an item from the queue to process it. We MUST call q.task_done() to indicate
            # that item is processed to prevent deadlock.
            while True:
                try:
                    # item = q.get()
                    q.get()

                    # TODO: We'll do work here.
                    log.info('Processed message')

                finally:
                    q.task_done()

def some_func():

    ...

    # Run in an infinite loop unless killed by user.
    try:
        log.info('Create pool with worker=%d to process messages', args.workers)
        with mp.Pool(processes=4) as pool:
            p = pool.apply_async(worker, (queue,))
            p.get()
        except KeyboardInterrupt:
            log.info('Received CTRL-C ... exiting')
            pass

        print('got here')

    return 0
user4979733
  • 3,181
  • 4
  • 26
  • 41

1 Answers1

1

Use asyncio, not multiprocessing

Depending on the nature of work to be done (whether CPU intensive or IO intensive) you might try asyncio, which has a simple pattern for graceful shutdown:

def main():
    queue = asyncio.Queue()
    loop = asyncio.get_event_loop()

    try:
        loop.create_task(publish(queue))
        loop.create_task(consume(queue))
        loop.run_forever()
    except KeyboardInterrupt:
        logging.info("Process interrupted")
    finally:
        loop.close()

^ from https://www.roguelynn.com/words/asyncio-graceful-shutdowns/

If you must use multiprocessing

This question has been answered here: Python multiprocessing : Killing a process gracefully

Eric
  • 158
  • 1
  • 6