1

I have been working on a problem for a while (I am a chemical engineer, so it takes me forever to understand how to code something) of how to have several tabs operating in their own process, but with each tab having it's own data to show in a matplotlib plot. I have been running into many pickling errors and I wondered if anyone had any somewhat simple solutions. I believe the main reasons for the pickling errors is due to the object I am trying to pass as a property into the tab object. This object holds some data as well as many other objects which help fit the data it holds. I feel like these objects are very nice and rather necessary, but I also realize that they are causing the problems with pickling. Here is a very simplified version of my code: (This will still compile if you want to copy/paste to test it out.)

import multiprocessing as mp
from PyQt4 import QtGui, QtCore
import numpy as np
import matplotlib
matplotlib.use('QtAgg')
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib import figure
import sys
import lmfit

# This object will just hold certain objects which will help create data objects ato be shown in matplotlib plots
# this could be a type of species with properties that could be quantized to a location on an axis (like number of teeth)
#, which special_object would hold another quantization of that property (like length of teeth) 
class object_within_special_object:
    def __init__(self, n, m):
        self.n = n
        self.m = m
    def location(self, i):
        location = i*self.m/self.n
        return location
    def NM(self):
        return str(self.n) + str(self.m)
# This is what will hold a number of species and all of their properties, 
# as well as some data to try and fit using the species and their properties
class special_object:
    def __init__(self, name, X, Y):
        self.name = name
        self.X = X
        self.Y = Y
        self.params = lmfit.Parameters()
        self.things = self.make_a_whole_bunch_of_things()
        for thing in self.things:
            self.params.add('something' + str(thing.NM()) + 's', value = 3)
    def make_a_whole_bunch_of_things(self):
        things = []
        for n in range(0,20):
            m=1
            things.append(object_within_special_object(n,m))
        return things
# a special type of tab which holds a (or a couple of) matplotlib plots and a special_object ( which holds the data to display in those plots)
class Special_Tab(QtGui.QTabWidget):
    def __init__(self, parent, special_object):
        QtGui.QTabWidget.__init__(self, parent)
        self.special_object = special_object
        self.grid = QtGui.QGridLayout(self)
        # matplotlib figure put into tab
        self.fig = figure.Figure()
        self.plot = self.fig.add_subplot(111)
        self.line, = self.plot.plot(self.special_object.X, self.special_object.Y, 'r-')
        self.canvas = FigureCanvas(self.fig)
        self.grid.addWidget(self.canvas)
        self.canvas.show()
        self.canvas.draw()
        self.canvas_BBox = self.plot.figure.canvas.copy_from_bbox(self.plot.bbox)
        ax1 = self.plot.figure.axes[0]
    def process_on_special_object(self):
        # do a long fitting process involving the properties of the special_object
        return
    def update_GUI(self):
        # change the GUI to reflect changes made to special_object
        self.line.set_data(special_object.X, special_object.Y)
        self.plot.draw_artist(self.line)
        self.plot.figure.canvas.blit(self.plot.bbox)
        return
# This window just has a button to make all of the tabs in separate processes
class MainWindow(QtGui.QMainWindow):
    def __init__(self, parent = None):
        # This GUI stuff shouldn't be too important
        QtGui.QMainWindow.__init__(self)
        self.resize(int(app.desktop().screenGeometry().width()*.6), int(app.desktop().screenGeometry().height()*.6))
        self.tabs_list = []
        central_widget = QtGui.QWidget(self)
        self.main_tab_widget = QtGui.QTabWidget()
        self.layout = QtGui.QHBoxLayout(central_widget)
        button = QtGui.QPushButton('Open Tabs')
        self.layout.addWidget(button)
        self.layout.addWidget(self.main_tab_widget)
        QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"), self.open_tabs)
        self.setCentralWidget(central_widget)
        central_widget.setLayout(self.layout)

    # Here we open several tabs and put them in different processes
    def open_tabs(self):
        for i in range(0, 10):
            # this is just some random data for the objects
            X = np.arange(1240.0/1350.0, 1240./200., 0.01)
            Y = np.array(np.e**.2*X + np.sin(10*X)+np.cos(4*X))
            # Here the special tab is created
            new_tab = Special_Tab(self.main_tab_widget, special_object(str(i), X, Y))
            self.main_tab_widget.addTab(new_tab, str(i))
            # this part works fine without the .start() function
            self.tabs_list.append(mp.Process(target=new_tab))
            # this is where pickling errors occur
            self.tabs_list[-1].start()
        return


if __name__ == "__main__":
    app = QtGui.QApplication([])
    win = MainWindow()
    win.show()
    sys.exit(app.exec_())

I have noticed that the errors are coming from matplotlib axes (I'm not sure how?) and gives the error pickle.PicklingError: Can't pickle <class 'matplotlib.axes.AxesSubplot'>: it's not found as matplotlib.axes.AxesSubplot. In addition, I have noticed that commenting out the matplotlib plots will also give the pickling error pickle.PicklingError: Can't pickle <function <lambda> at 0x012A2B30>: it's not found as lmfit.parameter.<lambda>. I think this is because lambda functions cannot be pickled and I guess lmfit has a lambda somewhere in it's depths... but I don't really know what to do without some way around these errors.

Oddly enough, the error I see form the original code (not the simplified version shown here) is slightly different, but still basically the same in sentiment. The error I get in my other code is pickle.PicklingError: Can't pickle 'BufferRegion' object: <BufferRegion object at 0x037EBA04>

Does anyone have a better solution to this problem by moving the objects around in regards to where I pass them or any other ideas?

I very much appreciate your time and effort and any help on this problem.

EDIT: I tried unutbu's idea in a way, but with some alterations to the position of the process funciton. The only problem with the proposed solution is that the do_long_fitting_process() function calls another function which iteratively updates the lines in the matplotlib plots. So the do_long_fitting_process() needs to have some access to the Special_Tab properties to change them and show the updates to the GUI.
I have tried doing this by pushing the do_long_fitting_process() function to just a global function and calling this:

[code] def open_tabs(self): for i in range(0, 10): ... self.tabs_list.append(new_tab)

        consumer, producer = mp.Pipe()
        process = mp.Process(target=process_on_special_object, args=(producer,))
        process.start()
        while(True):
            message = consumer.recv()
            if message == 'done':
                break
            tab.update_GUI(message[0], message[1])
        process_list[-1].join()

[/code] Where I am passing the data to update_GUI() via a mp.Pipe(), but the window just goes to "Not Responding" as soon as I start the processes.

chase
  • 3,592
  • 8
  • 37
  • 58

2 Answers2

1

The problem is that not all parts of an Axes object can be serialized, which is necessary to move data between the processes. I would suggest a slight re-organization of your code to push computation off to separate processes (that is any thing that will take more than a fraction of a second), but keep all of your plotting on the main process and only pass data back and forth.

Another option is to us QThread See time.sleep() required to keep QThread responsive? for two different ways of implementing it (one in the question, one in my answer).

Community
  • 1
  • 1
tacaswell
  • 84,579
  • 22
  • 210
  • 199
1

Separate the GUI code from the computation code. The GUI must run in a single process (though it may spawn multiple threads). Let the computation code be contained within in the special_object.

Let the Special_Tab call mp.Process when you want to perform a long-running computation:

class special_object:
    def do_long_fitting_process(self):
        pass    

class Special_Tab(QtGui.QTabWidget):    
    def process_on_special_object(self):
        # do a long fitting process involving the properties of the
        # special_object
        proc = mp.Process(target = self.special_object.do_long_fitting_process)
        proc.start()

class MainWindow(QtGui.QMainWindow):
    def open_tabs(self):
        for i in range(0, 10):
            ...
            self.tabs_list.append(new_tab)
            new_tab.process_on_special_object()
unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
  • Thank you for your answer! this seems like a great solution. I will try it out on the real code, and let you know if it works. – chase Feb 08 '13 at 20:24
  • The only problem with this is that the `do_long_fitting_process()` function iteratively updates the `matplotlib` plots. I have tried doing this by pushing the `do_long_fitting_process()` function to just a global function (see edit) but it is still causing problems – chase Feb 08 '13 at 21:11
  • 1
    @chase: Number One Rule: Keep the computation separate from the GUI. So `do_long_fitting_process` should not update the matplotlib plots. Instead when the computation is done, it should simply emit a signal. Take a look at [@tcaswell's code](http://stackoverflow.com/questions/14665636/time-sleep-required-to-keep-qthread-responsive/14666811#14666811). In particular, notice how he is using `QtCore.pyqtSignal` in `Worker` and calling `self.done.emit` method to signal when `Worker.get_data` ends. – unutbu Feb 08 '13 at 21:35
  • 1
    (cont'd): The `self.worker.done.connect(self.update_UI)` in `ApplicationWindow.__init__` causes `self.update_UI` to be called when the signal is emitted. Thus, the `ApplicationWindow` is in charge of all the GUI stuff, while the `Worker` is in charge of the computation. – unutbu Feb 08 '13 at 21:37
  • 1
    (correction): Rather, look at `self.pixel_list.emit(lE, hE)`, and how `self.worker.pixel_list.connect(self.update_figure)` causes `self.update_figure` to be called when the signal is emitted. – unutbu Feb 08 '13 at 22:07
  • Thanks unutbu, you have helped me out a great deal. I will look into signals and slots and try to update my work to fit with this schema. – chase Feb 09 '13 at 00:10
  • Just one more question regarding the architecture of OO programming. Are you saying that the way my classes are set up for `MainWindow` and `Special_Tab` is still capable of being on one main GUI thread as long as I don't pass a `special_object` to the `Special_Tab` as a parameter? Or do I need to put the `class Special_Tab` inside of the `class MainWindow` ? It doesn't matter where those classes are with regard to threading right? – chase Feb 09 '13 at 00:36
  • 1
    The GUI can be spread out among many classes, like `MainWindow` and `Special_Tab`. The way you have them set up is fine. Your `special_object` is sort of like @tcaswell's `Worker` class. – unutbu Feb 09 '13 at 00:45