Why can't matplotlib plot in a different thread?

Question

Minimum working example

I expect the following to show a plot, but i see no plot and the interpreter just hangs (my backend reports itself as TkAgg).

import matplotlib.pyplot as plt
from threading import Thread

def plot():
    fig, ax = plt.subplots()
    ax.plot([1,2,3], [1,2,3])
    plt.show()

def main():
    thread = Thread(target=plot)
    thread.setDaemon(True)
    thread.start()
    print 'Done'

How do I get the plot to display?

Context

I am running a simulation with lots iterations and would like to update my plot every 1000 iterations so that I can monitor how my simulation is evolving.

Psuedocode below:

iterations = 100000
for i in iterations:
    result = simulate(iteration=i)
    if not i % 1000:
        # Update/redraw plot here:
        # Add some lines, add some points, reset axis limits, change some colours

Having the plot in the main thread causes the plot GUI to hang/crash presumably because I have other work going on. So the idea was to do the plotting in a separate thread.

I have seen suggestions (e.g. here) to use a process rather than a thread. But then I cannot manipulate the figure or axes to add lines etc while my simulation runs because the figure object is in the remote process.

Edit

I'm not convinced this question is a duplicate of another one because that question deals with why the pyplot api cannot be used to manipulate two different plots that are each on a separate thread. It is because race conditions arising from executing two plots simultaneously prevents pyplot from determining which figure is the current figure.

However, I only have 1 plot and so pyplot only ever has a single and unique current figure.

Answer 1

As other people have told, Matplotlib is not thread safe, one option you have is to use multiprocessing. You say that this is not good for you, because you need access to the axes from different process, but you can overcome this by sharing data between the simulation process and the root process and then managing all the plotting related activities in the root process. For example

import matplotlib
matplotlib.use('TkAgg')
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg, NavigationToolbar2TkAgg
import multiprocessing
import time
import random
from Tkinter import *


#Create a window
window=Tk()



def main():
    #Create a queue to share data between process
    q = multiprocessing.Queue()

    #Create and start the simulation process
    simulate=multiprocessing.Process(None,simulation,args=(q,))
    simulate.start()

    #Create the base plot
    plot()

    #Call a function to update the plot when there is new data
    updateplot(q)

    window.mainloop()
    print 'Done'


def plot():    #Function to create the base plot, make sure to make global the lines, axes, canvas and any part that you would want to update later

    global line,ax,canvas
    fig = matplotlib.figure.Figure()
    ax = fig.add_subplot(1,1,1)
    canvas = FigureCanvasTkAgg(fig, master=window)
    canvas.show()
    canvas.get_tk_widget().pack(side=TOP, fill=BOTH, expand=1)
    canvas._tkcanvas.pack(side=TOP, fill=BOTH, expand=1)
    line, = ax.plot([1,2,3], [1,2,10])




def updateplot(q):
    try:       #Try to check if there is data in the queue
        result=q.get_nowait()

        if result !='Q':
             print result
                 #here get crazy with the plotting, you have access to all the global variables that you defined in the plot function, and have the data that the simulation sent.
             line.set_ydata([1,result,10])
             ax.draw_artist(line)
             canvas.draw()
             window.after(500,updateplot,q)
        else:
             print 'done'
    except:
        print "empty"
        window.after(500,updateplot,q)


def simulation(q):
    iterations = xrange(100)
    for i in iterations:
        if not i % 10:
            time.sleep(1)
                #here send any data you want to send to the other process, can be any pickable object
            q.put(random.randint(1,10))
    q.put('Q')

if __name__ == '__main__':
    main()

Answer 2

I had a similar problem where I wanted to update a mapltolib plot from a different thread, and I am posting my solution here in case others have a similar problem in the future.

As noted the tkagg are not threading safe so you must make sure all calls to matplotlib are from a single thread. This means that the threads must communicate, so that the 'plotting thread' always executes matplotlib functions.

My solution was to create a decorator, that will execute all decorated functions in the 'plotting thread', and then to decorate all the relevant functions. This allows you to do what you want without any change to syntax in the main code.

i.e. when you call ax.plot(...) in one thread, you will have it automatically executed in a different thread.

import matplotlib.pyplot as plt
import matplotlib
import threading
import time
import queue
import functools


#ript(Run In Plotting Thread) decorator
def ript(function):
    def ript_this(*args, **kwargs):
        global send_queue, return_queue, plot_thread
        if threading.currentThread() == plot_thread: #if called from the plotting thread -> execute
            return function(*args, **kwargs)
        else: #if called from a diffrent thread -> send function to queue
            send_queue.put(functools.partial(function, *args, **kwargs))
            return_parameters = return_queue.get(True) # blocking (wait for return value)
            return return_parameters
    return ript_this

#list functions in matplotlib you will use
functions_to_decorate = [[matplotlib.axes.Axes,'plot'],
                         [matplotlib.figure.Figure,'savefig'],
                         [matplotlib.backends.backend_tkagg.FigureCanvasTkAgg,'draw'],
                         ]
#add the decorator to the functions
for function in functions_to_decorate:
    setattr(function[0], function[1], ript(getattr(function[0], function[1])))

# function that checks the send_queue and executes any functions found
def update_figure(window, send_queue, return_queue):
    try:
        callback = send_queue.get(False)  # get function from queue, false=doesn't block
        return_parameters = callback() # run function from queue
        return_queue.put(return_parameters)
    except:
        None
    window.after(10, update_figure, window, send_queue, return_queue)

# function to start plot thread
def plot():
    # we use these global variables because we need to access them from within the decorator
    global plot_thread, send_queue, return_queue
    return_queue = queue.Queue()
    send_queue = queue.Queue()
    plot_thread=threading.currentThread()
    # we use these global variables because we need to access them from the main thread
    global ax, fig
    fig, ax = plt.subplots()
    # we need the matplotlib window in order to access the main loop
    window=plt.get_current_fig_manager().window
    # we use window.after to check the queue periodically
    window.after(10, update_figure, window, send_queue, return_queue)
    # we start the main loop with plt.plot()
    plt.show()


def main():
    #start the plot and open the window
    thread = threading.Thread(target=plot)
    thread.setDaemon(True)
    thread.start()
    time.sleep(1) #we need the other thread to set 'fig' and 'ax' before we continue
    #run the simulation and add things to the plot
    global ax, fig
    for i in range(10):
        ax.plot([1,i+1], [1,(i+1)**0.5])
        fig.canvas.draw()
        fig.savefig('updated_figure.png')
        time.sleep(1)
    print('Done')
    thread.join() #wait for user to close window
main()

Note that if you forget to decorate any functions, you may get a segmentation fault.

Also, in this example the child thread handles the plot and the main thread the simulation. In general it is advised to do the reverse, (i.e. let the main thread have the graphics).

Answer 3

I worked out a solution. Need no other UI library.

The following source opens a matplotlib window, whose canvas updates periodically.

import time
import _thread
import matplotlib.pyplot as plt
import numpy as np

def plotting_thread(fig, axe):
    while (True):
        mat = np.random.randn(256, 256)
        time.sleep(2)  # ... or some busy computing
        axe.clear()
        axe.imshow(mat)
        fig.canvas.draw_idle()  # use draw_idle instead of draw

fig = plt.figure()  # the figure will be reused later
axe = fig.add_subplot(111)

_thread.start_new_thread(plotting_thread, (fig, axe))

plt.show()

Three keys:

• Start a new thread for computing
• Use canvas.draw_idle to update your plot, which keeps the response of the window
• Use plt.show at the end of the program to start the GUI main event loop, which will block this program. So write no code after that.

Answer 4

The answer is taken here: https://matplotlib.org/stable/gallery/misc/multiprocess_sgskip.html I have tried several ways, but this is the only way that works for me.

    """
    Written by Robert Cimrman
    """

    import multiprocessing as mp
    import time

    import matplotlib.pyplot as plt
    import numpy as np

    np.random.seed(19680801)

    class ProcessPlotter(object):
        def __init__(self):
            self.x = []
            self.y = []

        def terminate(self):
            plt.close('all')

        def call_back(self):
            while self.pipe.poll():
                command = self.pipe.recv()
                if command is None:
                    self.terminate()
                    return False
                else:
                    self.x.append(command[0])
                    self.y.append(command[1])
                    self.ax.plot(self.x, self.y, 'ro')
                self.fig.canvas.draw()
            return True

        def __call__(self, pipe):
            print('starting plotter...')

            self.pipe = pipe
            self.fig, self.ax = plt.subplots()
            timer = self.fig.canvas.new_timer(interval=1000)
            timer.add_callback(self.call_back)
            timer.start()

            print('...done')
            plt.show()

    class NBPlot(object):
        def __init__(self):
            self.plot_pipe, plotter_pipe = mp.Pipe()
            self.plotter = ProcessPlotter()
            self.plot_process = mp.Process(
                target=self.plotter, args=(plotter_pipe,), daemon=True)
            self.plot_process.start()

        def plot(self, finished=False):
            send = self.plot_pipe.send
            if finished:
                send(None)
            else:
                data = np.random.random(2)
                send(data)


    def main():
        pl = NBPlot()
        for ii in range(10):
            pl.plot()
            time.sleep(0.5)
        pl.plot(finished=True)


    if __name__ == '__main__':
        if plt.get_backend() == "MacOSX":
            mp.set_start_method("forkserver")
        main()

Answer 5

The simplest answer probably is:

Because the backends aren't thread safe. Most GUI frameworks rely on calling "GUI" methods/functions from only one thread ("gui thread") and require more advanced methods when communicating with different threads ("worker threads").

You can find this in the documentation for Qt (PyQt/PySide), wxWidgets and (didn't find a more official source) Tkinter.

Answer 6

For people who find this from a search engine.

My solution was to use Threading.Lock() to still have the preprocessing steps computed with multiple threads but to only generate the Matplotlib part with a single thread.

lock.acquire()
plot_object.generate_plot()
lock.release()

Also, using mpl.use('Agg') help limiting interactive GUI stuff and better manage memory if one wants to generate hundreds of figures.

Here is an example with an in-house class:

import os
from threading import Thread, Lock
from multiprocessing import cpu_count
from queue import *

lock = Lock()

from PlotNIfTI.PlotNIfTI import PlotNifti


def worker_def(A):
    q = A
    while True:
        item = q.get()
        if item is None:
            break
        else:
            image_path, segmentation_paths, output_dir, segmentation_names, views = item
            try:
                plot_object = PlotNifti(image_path=image_path, segmentation_paths=segmentation_paths,
                                        show_contour=True, show_filled=True, transparency=0.20,
                                        get_at_centroid=True, segmentation_names=segmentation_names, crop_scan=True)
                for view in views:
                    plot_object.set_view(view)
                    plot_object.set_output_path(os.path.join(output_dir, 'screenshot_{}.png'.format(view)))
                    lock.acquire()
                    plot_object.generate_plot()
                    lock.release()
            except:
                print('     failed on {}'.format(image_path))
            q.task_done()

The entire class is publicly available here, if interested: https://github.com/guatavita/PlotNIfTI

Answer 7

Maplotlib is not thread safe. For me, trying to do something like an automated live update of a graph therefor is rather challenging with tkinter and matplotlib.

My workaround was to use the root.after function (tkinter: how to use after method)

Automatically live updates without requiring a separate thread :)

Also solves the issue where there is a no-error crash of tkinter when canvas.draw is called from a separate thread.