RAM full when creating plots in loops

Question

So I am just plotting a lot of plots, around 5000, and my RAM is running full. I've searched the site and web and found multiple solutions, such as plt.close, plt.clf() and gc.collect(), but neither help. I do not understand why my memory runs full even though I close all figures. Help would be much appreciated.

Same topic here but w/o a working solution for me: How can I release memory after creating matplotlib figures

y = range(5039*402) 
x = np.arange(0,402,1)
for i in xrange(len(data_collection)-1):
   plt.figure()
   plt.plot(x,y[i*402:402*(i+1)])
   plt.savefig('save%i.png'%(i))
   plt.close()

Sorry, might be an easy question, but I am new to python

Answer 1

You don't mention which version of matplotlib you're using, but this is a known issue with earlier versions (it's supposed to be fixed in 2.0.x). One method I've used to "get around" the memory leak is to create each plot in its own process. For example, you can use a subprocess, thread or multiprocess for each plot.

Python subprocess

Python threading

Python multiprocessing

My preferred approach is multiprocessing because (IMO) it's far easier to move things in and out of each process (objects must be pickleable.)

Python pickling

ETA: Here is a silly example to show how you might structure your script using multiprocessing.

import datetime as dt
import multiprocessing as mp

return_queue1 = mp.Queue()
return_queue2 = mp.Queue()

def foo(text, return_queue):
    for _ in range(5000):
        continue
    return_queue.put(dt.datetime.now())

def bar(text, return_queue):
    for _ in range(5000):
        continue
    return_queue.put(dt.datetime.now())

for _ in range(5):
    log = 0
    if __name__ == '__main__':
        for _ in range(100):
            p1 = mp.Process(name='p1', target=foo, args=('foo', return_queue1,))
            p2 = mp.Process(name='p2', target=bar, args=('bar', return_queue2,))
            p1.start()
            p2.start()

            if return_queue1.get() > return_queue2.get():
                log += 1

        p1.join()
        p2.join()
        print(u"Times bar won: {0}".format(log))