Python Loop memory leak

Question

I'm working on a bigger projekt and there i see a contiuous increase of memory usage. The code is similar to my demonstration sniped below - just not with random numbers and more usefull calculations... But the effect is the same. Running this i see increasing memory usage

Why is that happening and how can I prevent this? Thank you for every usefull answer :)

I'm using python 3

n= 10000000
indat = np.random.random(n*4)
data = np.zeros(n*3)

d = [0,0,0,0]
for i in range(n):  
    d = indat[i*4:i*4+4]
    data[i*3]  = np.sqrt(np.abs(d[0]+d[1]*d[3]))
    data[i*3+1]= np.sqrt(np.abs(d[3]+d[2]*d[3]))
    data[i*3+2]= np.sqrt(np.abs(d[2]+d[1]*d[3]))

EDIT: I tried some more stuff:

1: This works as expected - no memory increase

for i in range(n):  
    print(i, end="\r")

2: But with this there is that increase.

for i in range(n):  
    data[i] = indat[i]**2

So at minimum this simple copy is not working as i expect it to... How do I get python to store the result of my calculation in the already allocated memory?

Possible duplicate of [What is the difference between range and xrange functions in Python 2.X?](https://stackoverflow.com/questions/94935/what-is-the-difference-between-range-and-xrange-functions-in-python-2-x) — Yassine Faris, May 14 '18 at 15:41
I tested your code using python2.7, after replacing range by xrange. No problem. — ma3oun, May 14 '18 at 15:46

score 0 · Answer 1 · answered Dec 10 '18 at 17:54

Actually, that increasing memory usage happens because numpy optimizes the storage of arrays of zeros in memory. So, when you fill in that large array of zeros, numpy can't store it as efficiently.

Watch what happens to your memory usage if you create an array of zeros:

data = np.zeros(1000000000)

Then look at what happens when you create an array of ones:

data = np.ones(1000000000)

You'll see a dramatic increase in memory usage.

Python Loop memory leak

1 Answers1