Matplotlib and multiprocessing RuntimeError

Question

I'm trying to use multiprocessing and matplotlib together.

I'm creating a standard Pool, adding work with apply_async and updating the GUI with apply_async's callback function, which runs on the Pool's parent process (I verified this with os.getpid()). Example :

from pylab import *
from numpy import *
from numpy.random import random
from multiprocessing import Pool

# Output image
global out_all
out_all = zeros((256, 256))

# Only does something to in_image, doesn't access anything else
def do_work(in_image):
    for x in xrange(100000):
        out_image = in_image[::-1, ::-1]
    return out_image

# Update the output image and display if needed
def do_update(out_image):
    global out_all
    print ("Updating")
    out_all += out_image
    clf()
    imshow(out_all)
    show()

# Input images (close enough to what I do as well)
work = [random((256, 256)) for f in range(20)]

# Don't block when showing something
ion()

# Do the work
print "Starting pool"
pool = Pool()
for o in work:
    pool.apply_async(do_work, [o], callback=do_update).get()
pool.close()
pool.join()
print "Stopping pool"

# Block
ioff()
show()
print "Done"

The processing itself works fine, the processes are really destroyed on pool.join(), but Matplotlib (and TK, I guess) complain as soon as I try to do something afterwards, even only exiting the program :

Traceback (most recent call last):
  File "test_thread.py", line 27, in <module>
    show()
  File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 139, in show
    _show(*args, **kw)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 83, in __call__
    manager.show()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 444, in show
    self.canvas.draw_idle()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 258, in draw_idle
    self._idle_callback = self._tkcanvas.after_idle(idle_draw)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 512, in after_idle
    return self.after('idle', func, *args)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 504, in after
    name = self._register(callit)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1101, in _register
    self.tk.createcommand(name, f)
RuntimeError: main thread is not in main loop
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/_pylab_helpers.py", line 82, in destroy_all
    manager.destroy()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 452, in destroy
    self.canvas._tkcanvas.after_cancel(self.canvas._idle_callback)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 519, in after_cancel
    data = self.tk.call('after', 'info', id)
RuntimeError: main thread is not in main loop
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/_pylab_helpers.py", line 82, in destroy_all
    manager.destroy()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 452, in destroy
    self.canvas._tkcanvas.after_cancel(self.canvas._idle_callback)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 519, in after_cancel
    data = self.tk.call('after', 'info', id)
RuntimeError: main thread is not in main loop

My first thought was that the TK context was duplicated on each fork(), which somehow interferred with the TK loop in the main process, but I'm not doing anything TK-related in my workers. Any ideas?

Answer 1

The error messages reference Tkinter. So it looks like you are using the TkAgg backend. The code that follows is TkAgg/Tkinter-specific. In particular the call

win.after(100, animate)

makes use of the Tkinter-specific after method. There is an analogous call for GtkAgg/PyGtk, and similarly for other backends. But I just want to stress that what follows is TkAgg/Tkinter-specific.

---

Tkinter is intended to be run in a single thread. That is, all Tkinter GUI calls should originate from a single thread (usually, be not necessarily, the main thread).

The Pool's apply_async callback method runs in a separate (_handle_results) thread in the main process. Since imshow() is called from the Pool's _handle_results thread and show() is called in the main thread, Tkinter complains

RuntimeError: main thread is not in main loop

I don't see a way to use the apply_async callback in this situation.

Instead, what we can do is arrange for do_work to put out_image in a multiprocessing.Queue() (which I call out_queue in the code below). We'll then have the main process's main thread poll this Queue for items and display them as they come out of the Queue. This polling is done in the animate function, below.

---

plt.ion() is meant for interactive sessions only. Although it is sometimes possible to write little scripts which sort-of seem to work with plt.ion(), you'll get better results and cleaner GUIs if you resist using plt.ion() in scripts and instead write code that respects the GUI framework's event loop.

Although it is probably possible to fix your script and use plt.ion(), since this is not the recommended way of writing matplotlib scripts, let's see if we can avoid doing that.

---

plt.show() tells Tkinter to run its event loop. Notice that once this call is made, the GUI window is drawn, you can click buttons, zoom in and out, etc.

Somehow we need to inject a function into this event loop, to be run periodically by the event loop, and cooperatively with all the other GUI events that might be occurring. We want this function to check if any of our worker subprocesses have output for us, and if so, to update the imshow image.

With TkAgg/Tkinter, the way to inject such a function is

win = fig.canvas.manager.window
win.after(100, animate)

This will tell Tkinter to run the function animate (once) after (about) 100ms have elapsed. Since we want the function animate to run periodically, we just stick another

win.after(100, animate)

call at the end of animate.

---

import matplotlib as mpl
mpl.use('TkAgg')
import matplotlib.pyplot as plt
import numpy as np
import multiprocessing as mp
import logging
import Queue
logger = mp.log_to_stderr(logging.INFO)

# Only does something to in_image, doesn't access anything else
def do_work(in_image):
    logger.info('Processing in_image')
    for x in xrange(100000):
        out_image = in_image[::-1, ::-1]
    out_queue.put(out_image)

# Update the output image and display if needed
out_all = np.zeros((256, 256))


def pool_initializer(out_queue_):
    # Setup out_queue as a global variable *in the worker subprocesses*
    global out_queue
    out_queue = out_queue_


def animate():
    global out_all
    try:
        out_image = out_queue.get_nowait()
    except Queue.Empty:
        pass
    else:
        logger.info("Updating")
        out_all += out_image
        im.set_data(out_all)
        fig.canvas.draw()  # redraw the canvas
    win.after(100, animate)

if __name__ == '__main__':
    out_queue = mp.Queue()
    logger.info("Starting pool")
    pool = mp.Pool(initializer=pool_initializer, initargs=(out_queue, ))
    work = [np.random.random((256, 256)) for f in range(20)]
    for o in work:
        pool.apply_async(do_work, [o])
    pool.close()

    fig, ax = plt.subplots()
    win = fig.canvas.manager.window
    # Output image
    im = plt.imshow(out_all, vmin=0, vmax=1)

    # Register a function to be run once
    win.after(100, animate)
    plt.show()
    logger.info("Done")