3

I made very simple code in python, which I am running on jupyter:

# Cell 1
data = []
with open('test.txt', mode='r') as f:
    for line in f:
        data.append(line)

The file 'test.txt' has 3.1 Gb, and I can track the memory load.

Afther this, in the next cell, I exclude that:

# Cell 2
del data

But memory usage does not return to the initial state.

The next image is my system monitor. I executed the cell with the exclusion of the variable as soon as I finished the load, which reached almost 80%. But the memory only came back after I interrupted the notebook's kernel.

enter image description here

Isn't that wrong? I think the memory should have been released even on the jupiter, isn't it?

My system is Ubuntu 18.04, and I'm using miniconda, with Python 3.7.7.

smci
  • 32,567
  • 20
  • 113
  • 146
Klaifer Garcia
  • 340
  • 1
  • 12
  • 2
    I think it's IPython that causes this. It keeps a record of everything you evaluate in a dict `Out`. – wjandrea Oct 18 '20 at 01:35
  • 1
    Checking the result of [`gc.get_referrers`](https://docs.python.org/3/library/gc.html#gc.get_referrers) on `data` may be insightful. There are likely additional references to the list being stored in other parts of jupyter. – Brian61354270 Oct 18 '20 at 01:37
  • The title and body variously mention Python, iPython and jupyter; each of this will have slightly different memory behavior. If you claim it's an ipython issue not jupyter, then check you can reproduce it without running jupyter. Likewise, reproing Python and not ipython. – smci Oct 18 '20 at 03:31
  • In a simpler list append loop in `cell1`, I get the expected cycling of memory (as noted in the system monitor). As others note, separating OS, python, and jupyter/ipython issues can be tricky. And the interactive nature of a notebook means it's easy to mess up a sequence of cells runs. – hpaulj Oct 18 '20 at 03:46
  • I tried this code without Jupyter (I included a sleep at the end). The memory was freed without problems. On Jupyter, I also tried to repeat the execution. After the data variable was deleted, I started reading the file again. The memory volume has not changed, so I think the memory has been reused. – Klaifer Garcia Oct 18 '20 at 15:09
  • 1
    You can also use a magic `whos` to verify what variables there are in the interactive namespace. There is a certain amount of memory reuse; by some complicated logic python/numpy/juypter can choose to hang on to freed memory acticipating future use. – hpaulj Oct 18 '20 at 18:03

0 Answers0