How do I restructure this code so Pickle 'Buffer Region' errors do not occur

Question

I was wondering if anyone had any good solutions to the pickling error I am having at the moment. I am trying to set my code up to open several different processes in parallel, each with a fitting process to be display on a matplotlib canvas in real time. Within my main application, I have a button which activates this function:

def process_data(self):
        process_list = []
        for tab in self.tab_list:
            process_list.append(mp.Process(target=process_and_fit, args=(tab,)))
            process_list[-1].start()
            process_list[-1].join()
        return

As you may notice, a 'tab' (PyQt4.QtGui.QTabWidget object) is passed to the function process_and_fit, which I have noticed is not able to be pickled readily (link here) . However, I am not certain how to change the code to get rid of the frame being passed since it needs to be called in the process_and_fit function indirectly. By indirectly I mean something like this: (psuedo code again)

def process_and_fit(tab): # this just sets up and starts the fitting process
        result = lmfit.Minimizer(residual, parameters, fcn_args=(tab,))
        result.prepare_fit()
        result.leastsq()

def residual(params, tab):
    residual_array = Y - model
    tab.refreshFigure()
    return residual_array

class tab(QtGui.QTabWidget):
    def __init__(self, parent, spectra):
       # stuff to initialize the tab widget and hold all of the matplotlib lines and canvases

    # This just refreshes the GUI stuff everytime that the parameters are fit in the least squares method
    def refreshFigure(self):     
        self.line.set_data(self.spectra.X, self.spectra.model)
        self.plot.draw_artist(self.line)
        self.plot.figure.canvas.blit(self.plot.bbox)

Does anyone know how to get around this pickling error since the tab associated with a process should have only one set of data associated with it? I looked at Steven Bethard's approach but I really didn't understand where to put the code or how to utilize it. (I am a chemical engineer, not a computer scientist so there's a lot that I don't understand)

Any help is greatly appreciated.

EDIT: I added the links in that I forgot about, as requested.

Answer 1

The main issue is that you can't make UI changes from a separate process from the main UI thread (the one that all of your Qt calls are in). You need to use a mp.Pipe or mp.Queue to communicate back to the main process.

def process_data(self):
    for tab in self.tab_list:
        consumer, producer = mp.Pipe()
        process_list.append(mp.Process(target=process_and_fit, args=(producer,)))
        process_list[-1].start()
        while (true):
            message = consumer.recv()  # blocks
            if message == 'done':
                break
            # tab.spectra.X, tab.spectra.model = message
            tab.refreshFigure()
        process_list[-1].join()
    return

def process_and_fit(pipe_conn):
    ...
    pipe_conn.send('done')

def residual(params, pipe_conn):
    residual_array = Y - model
    pipe_conn.send('refresh')  # or replace 'refresh' with (X, model)
    return residual_array

One more thing to note: blocking for the consumer.recv() will probably hang the GUI thread. There are plenty of resources to mitigate this, the question "subprocess Popen blocking PyQt GUI" will help, since you should probably switch to QThreads. (Qthread: PySide, PyQt)

The advantage of using QThreads instead of Python threads is that with QThreads, since you're already in Qt's main event loop, you can have asynchronous (non-blocking) callbacks to update the UI.