xPython savefig memory leak unresolved with formerly proposed solutions

Question

EDIT: SOLVED

But i'm not sure how... i moved as much as I could to pylab instead of pyplot, implemented a multiprocessing approach described [here][1], and at that point it was working fine. Then in an attempt to pinpoint the issue, i reversed my code step-by-step. But it kept working, even without the multiprocessing, even with pyplot, etc. Only when i take off fig.clf() now it does not work, which is what most people seem to experience but was not the issue with me initially. Well maybe it was, maybe the clf() statement wasnt at the right place or something. Thanks !

EDIT: ISSUE STILL NOT SOLVED

That's very surprising. I now moved my savefig() function into an external module, which I import when executing my script. Here is the function:

def savethefig(fig,outpath,luname):
    plt.savefig(outpath+luname+'testredtarget.png',dpi=500)

So now i do something like:

from myfile import savethefig
fig = plt.figure()
ax1 = fig.add_subplot(311)
pftmap=zeros(shape=(1800,3600))*nan
    for i in range(len(pftspatialnames)):
       pftmap[indlat,indlon]=data[i,:]
       fig = pylab.gcf()
       a1=ax1.imshow(pftmap, interpolation='nearest',origin='upper', aspect='auto')
       savethefig(fig,outpath,luname)

I did everything step by step, line by line and the RAM definitely goes up when hiting the savefig() function within the external function. Goes up by about 500MB. Then when going back to the main script, that memory is not released. Aren't external function supposed to clear everything ? I'm missing something...

ORIGINAL POST:

I'm using python EDP 7.3-2 (32 bit) (python 2.7.3).

I have a program that does some computations, then maps some of the results out and saves the images, with matplotlib.

That's a pretty large amount of data, and if i try to map and save too many images, I hit a memory limit. Which I shouldnt since I re-use the same figure, and dont create new variables. I've struggled for a while with that, tried all the figure clearing/deleting etc solutions, changing the backend used by matplotlib, but nothing does it, everytime the code hits the savefig function, it adds a lot of memory and does not take it off later.

I'm far from being an expert on memory stuff (or python by the way), but here is one diagnose attempt i ran, using the module objgraph:

from numpy import *
import matplotlib
matplotlib.use('Agg')
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib import pyplot as plt

outpath = '/Users/yannick/Documents/outputs/'
pftspatialnames = ['forest','shrubs','grass']

Computations (not shown)

fig = plt.figure()
for i in range(len(pftspatialnames)):
   pftmap=zeros(shape=(1800,3600))*nan
   pftmap[indlat,indlon]=data[i,:]
   fig = pylab.gcf()
   ax1 = plt.subplot2grid((3,1),(0,0))
   a1=ax1.imshow(pftmap, interpolation='nearest',origin='upper', aspect='auto')
   del(pftmap)
   gc.collect()

   print 'MEMORY LEAK DETECTOR before figsave'
   objgraph.show_growth()

   plt.savefig(outpath+pftspatialnames[i]+'testredtarget.png', dpi=500)

   print 'MEMORY LEAK DETECTOR after figsave'
   objgraph.show_growth()

Pretty big maps (and there's 3 of them, 3 subplots, just showed one here), but it handles it pretty well. It takes about 4 figures to saturate memory. Some stuff may be superfluous (e.g. deleting pftmap), but i was trying everything to clear some memory.

And here is the printed output:

MEMORY LEAK DETECTOR before figsave
dict                6640     +2931
weakref             3218     +1678
instance            1440     +1118
tuple               3501      +939
function           12486      +915
Bbox                 229      +229
instancemethod       684      +171
Line2D               147      +147
TransformedBbox      115      +115
Path                 127      +114
MEMORY LEAK DETECTOR after figsave
dict                7188      +548
Path                 422      +295
weakref             3494      +276
instance            1679      +239
tuple               3703      +202
function           12670      +184
TransformedPath       87       +87
instancemethod       741       +57
Line2D               201       +54
FontProperties       147       +36

So before calling savefig there's a lot of new objects (that's normal i do a bunch of stuff in the code before). But then by just calling savefig we add 550 dict, etc. Could that be the source of my problem ? Note that this happens the first time i call savefig only. Any subsequent call does the following:

MEMORY LEAK DETECTOR before figsave
MEMORY LEAK DETECTOR after figsave
tuple        3721        +6
dict         7206        +6
function    12679        +3
list         2001        +3
weakref      3503        +3
instance     1688        +3
Bbox          260        +3

but memory still keeps growing and soon i hit memory limit.

Thanks a lot !

The code path in saving depends on exactly what artists in your figure. Please provide complete (but minimal) code to demonstrate the problem. — tacaswell, Nov 19 '13 at 01:58
I dont really understand what "artists" means, but i've added some more details on the path, figure construction, etc. Thanks ! — le page yannick, Nov 19 '13 at 02:14
Have you tried the solutions found [here](http://stackoverflow.com/questions/7101404/how-can-i-release-memory-after-creating-matplotlib-figures)? — rickhg12hs, Nov 19 '13 at 07:58
I did yes (including pyplot.close()). Also i have my plotting into a function, which i thought would release the memory once back to the main program, but that's not the case. I'm now trying the solution proposed with Canvas to bypass pyplot... Thanks — le page yannick, Nov 19 '13 at 14:55

score 0 · Answer 1 · answered Nov 19 '13 at 08:28

0

Googled anecdotal evidence seems to suggest that executing 'matplotlib.pyplot.close' after a savefig will reclaim/free the memory associated with the figure. See pyplot docs for all the calling options.

answered Nov 19 '13 at 08:28

rickhg12hs

10,638
6
24
42

score 0 · Answer 2 · answered Nov 19 '13 at 08:28

0

I forgot which stack overflow thread/website I found this solution at, but using the figure directly (and not touching the pyplot state machine) usually does the trick. So:

fig = figure.Figure()
canvas = FigureCanvas(fig)
...
canvas.print_figure(outpath+pftspatialnames[i]+'testredtarget.png', dpi=500)

answered Nov 19 '13 at 08:28

story645

566
3
13

Thanks, I tried that and saw it was successful to some for memory issues. In my case it didnt help. Finally found a solution, posted in the original message. – le page yannick Nov 19 '13 at 20:25

score 0 · Accepted Answer · answered Nov 20 '13 at 21:11

EDIT: SOLVED

But i'm not sure how... i moved as much as I could to pylab instead of pyplot, implemented a multiprocessing approach described [here][1], and at that point it was working fine. Then in an attempt to pinpoint the issue, i reversed my code step-by-step. But it kept working, even without the multiprocessing, even with pyplot, etc. Only when i take off fig.clf() now it does not work, which is what most people seem to experience but was not the issue with me initially. Well maybe it was, maybe the clf() statement wasnt at the right place or something. Thanks

xPython savefig memory leak unresolved with formerly proposed solutions

3 Answers3