0

I wrote a function that logged the process using logging python package and redirected it to both a file and the ipython console (I'm using Spyder). After a couple of hours, I noticed that the memory increased slowly but surely.

Before the loop, here is my logging initialization block:

##### Initialize logging

log_file='program.log'
logger = logging.getLogger()
logger.handlers=[]
logger.setLevel(logging.INFO)
fh= logging.FileHandler(log_file,mode='w')
sh= logging.StreamHandler()
logger.addHandler(fh)
logger.addHandler(sh)

Then it the main loop I do couples of: 

logging.info('Some text')

How can I avoid those memory leaks?

ChrisB
  • 123
  • 1
  • 3
  • 13
  • how are you judging memory usage? have are you sure it's not just ipython saving everything into the magic `In` and `Out` variables (which can be cleared with `.clear()` – Sam Mason Sep 28 '18 at 20:20
  • To check the memory usage I'm simply using 'open system monitor' on my ubuntu and observe that the kernel use more and more memory. I cannot use `clear()` as some variables are defined outside the loop. `del` variables that are inside the loop do not change the memory usage. – ChrisB Sep 28 '18 at 21:26
  • `Out` is a `dict`, mostly maintained by ipython, and it saves the output of every block of code executed. maybe try running your code in a vanilla python interpreter that doesn't do this, depending on your code it might help – Sam Mason Sep 28 '18 at 22:02
  • I didn't know the existence of `In` and `Out` thanks for the tip! However these `dict` only store strings showing commands being executed in the ipython console, they don't grow in loops and are not taking 1Gb of my RAM. – ChrisB Sep 28 '18 at 22:29
  • I've just tried putting that code into a file, and a infinite loop that logs text every 10ms. I've left it running for a while and RAM usage is stable. I'd suggest that something other than the `logging` module causing a memory leak… – Sam Mason Oct 01 '18 at 10:53
  • as an aside, the `Out` variable contains normal references to python objects, not just string representations of them. hence any memory they refer to will be effectively pinned by the garbage collector – Sam Mason Oct 01 '18 at 10:55
  • So I re-run the process and added `In.clear()` and `Out.clear()` in the loop, memory usage is still increasing... I might need some more time to identify the source of it. – ChrisB Oct 01 '18 at 16:55
  • 1
    I'm pretty sure it isn't to do with IPython or logging, maybe one of https://mg.pov.lt/objgraph/ or https://stackoverflow.com/a/1641280/1358308 would help – Sam Mason Oct 02 '18 at 13:38

0 Answers0