60

UPDATE:- This problem solved itself after a machine reboot. Not yet able to figure out why this error was happening before.

I have a function that loads a huge numpy array (~ 980MB) and returns it.

When I first start Ipython and call this function, it loads the array into the variable without any problem.

But if I run the same command again, it exits raising a "Memory Error".

I tried the following,

del hugeArray

Still the same error was occurring. I even tried the following

del hugeArray
gc.collect()
gc.collect()

Initially, gc.collect() returned 145 and the second call returned 48. But even after this when I call the function, it was still raising a Memory error.

The only way I could load again was to restart ipython. Is there something I can do to free all memory in ipython, so that I don't have to restart it?

----------------Update

Following is the output of %whos

Variable   Type      Data/Info
------------------------------
gc         module    <module 'gc' (built-in)>
gr         module    <module 'Generate4mRamp' <...>rom 'Generate4mRamp.pyc'>
np         module    <module 'numpy' from '/us<...>ages/numpy/__init__.pyc'>
plt        module    <module 'matplotlib.pyplo<...>s/matplotlib/pyplot.pyc'>

Out of this, gr is my module containing the function which i used to load the data cube.

---------How to Reproduce the error

The following simple function is able to reproduce the error.

import numpy as np
import gc

def functionH():
    cube=np.zeros((200,1024,1024))
    return cube

testcube=functionH()   #Runs without any issue 

del testcube
testcube=functionH()  # Raises Memory Error

del testcube
gc.collect()
gc.collect()
testcube=functionH()  # Still Raises Memory Error

This error is occurring only in Ipython. In simple python (>>>) after giving del testcube, there is no Memory Error.

Saullo G. P. Castro
  • 56,802
  • 26
  • 179
  • 234
indiajoe
  • 1,291
  • 3
  • 13
  • 26
  • 1
    Can you try calling `whos` in ipython to find out what is taking up memory? – tiago Apr 28 '13 at 09:15
  • @tiago : I have added the output of the %whos command. It doesn't show any thing other than the modules i loaded. – indiajoe Apr 28 '13 at 09:52
  • Can you show the reference count of the object before delete it? `import sys;sys.getrefcount(testcube)` – HYRY Apr 28 '13 at 11:34
  • sys.getrefcount(testcube) gave me output 2 – indiajoe Apr 28 '13 at 11:48
  • Hey, All of a sudden it started working now. after del command it is able to run the function again without Memory Error. I shall try to see what change happened in between. – indiajoe Apr 28 '13 at 11:55
  • This is very strange, now everything is working fine. I had rebooted the machine and also had started a pure python interface, apart from that everything else was same. Now I'm not able to reproduce any of the errors I mentioned in the question... – indiajoe Apr 28 '13 at 12:13
  • what do you mean by 'all of a sudden it started working..'? Could you explain in detail coz I've also encountered this problem in Ipython notebook – LittleLittleQ Dec 11 '14 at 12:12
  • @AnnabellChan It all started working fine after a system reboot, I don't know what happened. I was no longer able to reproduce the error. It could have been some OS level problem. – indiajoe Dec 11 '14 at 23:21

1 Answers1

62

Are you looking at the value? IPython caches output variables as e.g. Out[8], so if you examine it, it will be kept in memory.

You can do %xdel testcube to delete the variable and remove it from IPython's cache. Alternatively, %reset out or %reset array will clear either all your output history, or only references to numpy arrays.

Thomas K
  • 39,200
  • 7
  • 84
  • 86
  • I wasn't looking at the value in IPython. But thanks for this info about the Ipython caching the outputs. I didn't know about this. I shall try out this xdel command also when this error occurs next time. Right now, everything is working fine. – indiajoe Apr 29 '13 at 13:18
  • 4
    Is there a reason for this not to work properly? I try doing something like `list([i**2 for i in range(30000000)])` to test this, and then I try to do the %reset out. Out is empty after that, but the memory is still occupied as before the reset command, why could that be? It's like it's only removing the reference. but not releasing memory? – Kobe-Wan Kenobi Feb 04 '17 at 14:49
  • 1
    I should add that %reset, without `out` does the work. It both empties the memory and removes the references. But I don't want to loose my own variables. Also, this works `v = Out[18] %xdel v`. Why is that? – Kobe-Wan Kenobi Feb 04 '17 at 14:51
  • 2
    `%xdel` tries to find and remove all the references to an object in IPython's machinery. – Thomas K Feb 05 '17 at 17:06