2

How does Python deal with memory allocation? Is there anything relevent about memory allocation that I, as a developer, would need to think about while creating a large application using Python, or should I trust Python to manage this in the most optimal way?

Edward Haigh
  • 43
  • 7
  • 1
    [Read the docs](https://docs.python.org/3/c-api/memory.html) – Cory Kramer May 17 '16 at 12:15
  • The docs say that the "Python Memory Manager" manages the memory but the don't answer the question (or at least not in that section) as to whether there is anything that a developer needs to think about. I guess the key things will be around making sure redundant objects are able to be garbage collected etc. I'll leave answering the question to someone who knows rather more about this than I do. – neil May 17 '16 at 12:22
  • Neil is right, I had read the docs but I was left unsure if I should be doing anything myself. – Edward Haigh May 17 '16 at 12:34
  • 2
    Great question. I look forward to read an elaborate answer to this from one of the Python veterans here myself. As far as I know, with Python like with every other language, you've got the CPU cache levels to take advantage of by organizing your loops and by trying to pack object members close to each other according to their access order. – Aiman Al-Eryani May 17 '16 at 12:52
  • You might want to read: [CPython memory allocation](http://stackoverflow.com/q/18522574/846892) – Ashwini Chaudhary May 17 '16 at 13:21

1 Answers1

5

Python handles allocation for you, but there are circumstances where you should explicitly deallocate to stop Python or the wider system from running short on memory. del x removes the name x from whatever data it is attached to, and if the data reference count becomes zero, then that data is eligible for garbage collection when the garbage collector is next invoked. Which is certain to have happened before the program fails because it has run out of memory.

You can explicitly invoke the garbage collector (standard Python module gc). gc.collect() tells it to do a full collection immediately if you do not want it to happen at a random later time. There are a number of other things you can do to tune its behaviour to the needs of your program. For example, you can disable and re-enable it around a section of code whose realtime behaviour is important and whose churn of temporary objects is slight. See https://docs.python.org/3/library/gc.html for more info.

Garbage collection of an object may be when important tidying-up of external entities takes place. The simplest example is f=open("some.file" ,"w"). If you do not use with or invoke f.close() the file stays open until program termination or until f is garbage-collected.

Do you need to know this? Often not on a single-user system with Gbytes to spare, like the average desktop PC. Probably yes, if you are writing Python code to run as one element of a collection of processes on a hard-pressed server. The memory that you don't explicitly deallocate, may be memory that some other process needs.

nigel222
  • 7,582
  • 1
  • 14
  • 22