Python multiprocessing+savefig leads to error or system lockup

Question

I have a big 3d numpy array, each slice (2d array) of I want to write out to an imshow-like figure (i.e. a heatmap of the values). As a concrete example, say the array is of shape 3x3x3000, so I want 3000 images, each of which represents a 3x3 matrix. Looping over it with a single thread is a bit slow. Since the iterations are completely independent, I thought to use the multiprocessing module to speed it up a bit. Code is as below.

def write_tensor_image(t_slice_wrapper):

    idx = t_slice_wrapper['idx']
    t_slice = t_slice_wrapper['t_slice']
    folder_path=t_slice_wrapper['folder_path']

    fig = matplotlib.pyplot.figure()
    ax = fig.add_subplot(111)    
    ax.imshow(t_slice,interpolation='none')
    fig.tight_layout()

    fname_ = os.path.join(folder_path,'tmp_%s.png'%str(idx))
    fig.savefig(fname_, bbox_inches="tight")    

def write_tensor_image_sequence(tensor, folder_path='/home/foo/numpy_cache'):

    os.system('mkdir -p %s'%folder_path)
    os.system('rm -rf %s/*'%folder_path)

    slices = [None]*tensor.shape[2]
    for i in range(0,tensor.shape[2]):
        slices[i] = {'t_slice':tensor[:,:,i], 'idx':i, 'folder_path':folder_path}

    pool = multiprocessing.Pool(processes=4)
    pool.map(write_tensor_image, slices)
    pool.close()
    pool.join()

However this does not work - the single threaded case works fine (just calling write_tensor_image() in the for-loop) but using the pool either causes total lockup of the machine or gives something like the following error:

XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
  after 849 requests (849 known processed) with 28 events remaining.
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
  after 849 requests (849 known processed) with 28 events remaining.
  after 849 requests (849 known processed) with 28 events remaining.
X Error of failed request:  BadPixmap (invalid Pixmap parameter)
Major opcode of failed request:  54 (X_FreePixmap)
Resource id in failed request:  0x4e0001e
Serial number of failed request:  851
Current serial number in output stream:  851

I think I'm on the right track (according to e.g. How to fix the python multiprocessing matplotlib savefig() issue? and Matplotlib: simultaneous plotting in multiple threads), but I must be missing something.

score 4 · Accepted Answer · answered Mar 06 '15 at 17:15

4

Do a matplotlib.use('agg') at the head of the script. Matplotlib appears to be trying to establish a GUI in each sub-process, which are conflicting with one-another.

More generally, you may not want to use the pyplot interface, but rather the OOP one, in cases where you are not doing standard interactive plotting.

answered Mar 06 '15 at 17:15

mdurant

27,272
5
45
74

Hmm, just tried this (also capitalized as suggested here http://stackoverflow.com/questions/4931376/generating-matplotlib-graphs-without-a-running-x-server), but still generates the error. Also tried matplotlib.pyplot.ioff(). No difference :( Searching for OOP (object oriented?) interface to matplotlib didn't turn up much. This is my first 48 hours of numerical development in python (previously Matlab), so I don't really know the nuances yet. – normally_deviated Mar 06 '15 at 17:37
Solved: I had imported pyplot in another module, which I turn imported before setting the matplotlib backend. So matplotlib was saying that the matplotlib.use() call had no effect. Works now. Thanks! – normally_deviated Mar 06 '15 at 17:50

Python multiprocessing+savefig leads to error or system lockup

1 Answers1