2

Here is my test code for python3:

#! /usr/bin/python3
from memory_profiler import profile

@profile(precision=10)
def f():
    huge_list = [x for x in range(2000)]
    del huge_list
    print("finish")

if __name__ == "__main__":
    f()

output:

Line #    Mem usage    Increment   Line Contents
================================================
     4  17.2109375000 MiB  17.2109375000 MiB   @profile(precision=10)
     5                             def f():
     6  17.2109375000 MiB   0.0000000000 MiB       huge_list = [x for x in range(2000)]
     7  17.2109375000 MiB   0.0000000000 MiB       del huge_list
     8  17.2226562500 MiB   0.0117187500 MiB       print("finish")

It shows that huge_list = [x for x in range(2000)] doesn't take any memory.
I changed it to huge_list = "aa" * 2000, it's the same.
But if I change 2000 to 20000, it gets some memory.

why ?

A similar question is here: What does “del” do exactly?

olivetree123
  • 173
  • 3
  • 13
  • See [How does Memory Allocation work in Python (and other languages)](https://medium.com/datadriveninvestor/how-does-memory-allocation-work-in-python-and-other-languages-d2d8a9398543) – Shivam Jha Oct 14 '20 at 07:05
  • See https://stackoverflow.com/questions/7247298/size-of-list-in-memory – Thierry Lathuille Oct 14 '20 at 07:05
  • No, it's different – olivetree123 Oct 14 '20 at 08:24
  • From the doc of the library you used: "Q: How accurate are the results ? A: This module gets the memory consumption by querying the operating system kernel about the amount of memory the current process has allocated, which might be slightly different from the amount of memory that is actually used by the Python interpreter. Also, because of how the garbage collector works in Python the result might be different between platforms and even between runs." – Thierry Lathuille Oct 14 '20 at 09:04
  • You should consider using [tracemalloc](https://docs.python.org/3.8/library/tracemalloc.html#module-tracemalloc) if you want precise memory tracing of your app. – Lescurel Oct 14 '20 at 09:17
  • I have tried it in win10 64bit and ubuntu 20.04 64bit，they have the same result – olivetree123 Oct 14 '20 at 12:00

1 Answers1

2

I am not sure how exactly things work but I think this is what happens:

  • The memory-profiler does not measure the memory that is actually used by the interpreter, but of the whole process, as stated in the FAQ of memory-profiler:

Q: How accurate are the results ?
A: This module gets the memory consumption by querying the operating system kernel about the amount of memory the current process has allocated, which might be slightly different from the amount of memory that is actually used by the Python interpreter. Also, because of how the garbage collector works in Python the result might be different between platforms and even between runs.

  • if the interpreter reserved enough memory beforehand, so that the new list fits in the free reserved memory, then the process does not need more memory to reserve. So memory-profiler correctly prints out a change of 0. A list of some 1000 integers is not really huge after all, so I'd not be surprised if the interpreter usually has some free reserved memory dangling around for that.
  • if the required memory for the new huge list does not fit in the already reserved memory, the process needs to reserve more, which the memory-profiler actually sees.
  • abarnert's answer to What does “del” do exactly? is basically saying the same thing.
  • despite all that, the results on my machine are different (using Python 3.8.5, 64 bit (AMD64) on win32 and memory-profiler 0.57.0:
     Line #    Mem usage    Increment   Line Contents
     ================================================
     4  41.3984375000 MiB  41.3984375000 MiB   @profile(precision=10)
     5                             def f():
     6  41.4023437500 MiB   0.0039062500 MiB       huge_list = [x for x in range(1)]
     7  41.4023437500 MiB   0.0000000000 MiB       del huge_list
     8  41.4101562500 MiB   0.0078125000 MiB       print("finish")

So I guess it depends on the system very much ....

EDIT:

As Lescurel wrote in the questions comments:

You should consider using tracemalloc if you want precise memory tracing of your app.

Ch L
  • 76
  • 7