5

I need to export multiple plots with Python and Matplotlib but I have issues with the memory management.

I'm using Jupyter Notebook in Windows and the code is structured as shown in the working example here below. Basically, I'm generating a daily plot over a period of several years.

Following the memory usage I can see that usually for the first 4 "months" the memory usage is fairly stable but then it increases more than 15MB for each "day". With the complete code this leads to a memory error.

I tried to use plt.close() as suggested here. I also tried plt.close(fig), plt.close('all') and to manually delete some variables with del. Finally I've put gc.collect in the loop. Nothing seems to have any effect.

import numpy as np
import matplotlib.pyplot as plt
import gc

sz = 1000
i=0
for year in [2014,2015,2016,2017]:
    for month in [1,2,3,4,5,9,10,11,12]:
        for day in range(1,30):

            # Plot
            fig, ((ax1,ax2),(ax3,ax4)) = plt.subplots(2,2,figsize=(14,10))

            x1 = np.arange(1,sz+1,1)
            y1 = np.random.random(sz)
            ax1.scatter(x1,y1)
            ax2.scatter(x1,y1)
            ax3.scatter(x1,y1)
            ax4.scatter(x1,y1)

            filename = 'test/test_output{}{}{}.png'.format(year,month,day)
            plt.tight_layout()
            plt.savefig(filename, bbox_inches='tight')

            #plt.close(fig)            
            #plt.close('all')
            plt.close()

            del x1, y1, ax1, ax2, ax3, ax4, fig

            # counter
            i=i+1

        # Monthly operations
        print(filename,i)
        gc.collect()

# Triggered outside the loops
gc.collect()

On the other hand, if I interrupt the code (or I wait for the "Memory error" to occur) and then I run manually gc.collect, the memory is cleaned right away.

Why gc.collect does not have any effect in the nested loops?

Is there a way to improve the memory management for this code?

Davide
  • 175
  • 4
  • 12
  • What happens if you run as a script instead of in a notebook? I can imagine her notebook doing funky things when given hundreds of plots even if you close them. – Jody Klymak Jul 23 '18 at 14:32
  • I tried to run the script with Spyder and it happens the same (Python 3.6.4 - Spyder 3.2.6 - IPython 6.2.1). – Davide Jul 23 '18 at 14:58
  • 1
    "As script" means *outside* of IPython or similar, i.e. in the console run `python scipt_name.py`. How do you measure memory usage? – ImportanceOfBeingErnest Jul 23 '18 at 16:54
  • 1
    Ok, I tried to run the code as a script and the memory usage is kept correcly under control. It seems that `plt.close()` works as expected in this case. So the problem is with IPython and I still cannot find a solution. For checking the memory usage here I simply watch the Task Manager. – Davide Jul 24 '18 at 17:38
  • 1
    What happens if you turn interactive mode off, `plt.ioff()` when in IPython? – ImportanceOfBeingErnest Jul 28 '18 at 11:08

0 Answers0