How to dynamically allocate memory in Python

Question

Is there any method in python, that I can use to get a block of memory from the heap ,and use a variable to reference it. Just like the keyword "new" , or the function malloc() in other languages:

Object *obj = (Object *) malloc(sizeof(Object)); 
Object *obj = new Object();

In the project, my program is waiting to receive some data in uncertain intervals and with a certain length of bytes when correct.

I used to it like this:

void receive()// callback
{
  if(getSize()<=sizeof(DataStruct))
  {
   DataStruct *pData=malloc(sizeof(DataStruct));
   if(recvData(pData)>0)
   list_add(globalList,pData);
  }   
}

void worker()
{
  init()
  while(!isFinish)
 {
  dataProcess(globalList);
 }

}

Now, I want to migrate these old project to python, and I tried to do it like this:

def reveive():
 data=dataRecv()
 globalList.append(data)

However, I get the all item in the list are same, and equal to the latest received item. It is obvious that all the list items are point to the same memory adress, and I want to get a new memory adress each the function is called.

You are not trying to solve the problem in the right way. Python is a higher-level language, you don't allocate memory at all. If you show the `dataRecv` definition (or whatever that then calls - a [mcve], please) it might be clearer what the *actual problem* is. My guess is you're emptying and reusing a single list object rather than creating a new one. Possibly you have the dreaded [mutable default argument](http://stackoverflow.com/q/1132941/3001761). — jonrsharpe, Jun 24 '16 at 08:43
thanks,dataRecv is from another import module,and I omit the argument that used to call dataRecv.I print data in reveive() ,it is right.As you say,I am not creating a new list object.this is just what I ask ,I need a method to achieve the keyword "new" in c++ — lb lb, Jun 24 '16 at 09:04

score 8 · Answer 1 · answered Jun 28 '16 at 07:27

The equivalent of "new" in python is to just use a constructor eg:

new_list = list() # or [] - expandable heterogeneous list
new_dict = dict() # expandable hash table
new_obj = CustomObject() # assuming CustomObject has been defined

Since you are porting from C, some things to note. Everything is an object in python including integers, and most variables are just references, but the rules for scalar variables such as integers and strings are different from containers, eg:

a = 2   # a is a reference to 2
b = a   # b is a reference to 'a'
b = 3   # b now points to 3, while 'a' continues to point to 2

However:

alist = ['eggs', 2, 'juice']  # alist is reference to a new list
blist = alist # blist is a reference; changing blist affects alist
blist.append('coffee') # alist and blist both point to 
                       # ['eggs', 2, 'juice', 'coffee']

You can pre-allocate sizes, if you'd like but it often doesn't buy you much benefit in python. The following is valid:

new_list4k = [None]*4096 # initialize to list of 4096 None's 
new_list4k = [0]*4096    # initialize to 4096 0's
big_list = []
big_list.extend(new_list4k) # resizes big_list to accomodate at least 4k items

If you want to ensure memory leaks do not occur, use local variables as often as possible, eg, within a function so as things go out of scope you don't have to worry.

For efficient vectorized operations (and much lower memory footprint) use numpy arrays.

import numpy as np
my_array = np.zeros(8192) # create a fixed array length of 8K elements
my_array += 4             # fills everything with 4

My added two cents: I'd probably start by asking what your primary goal is. There is the pythonic ways of doing things, while trying to optimize for speed of program execution or minimum memory footprint. And then there is the effort of trying to port a program in as little time as possible. Sometimes they all intersect but more often, you will find the pythonic way to be quick to translate but with higher memory requirements. Getting higher performance out of python will probably take focused experience. Good luck!

Thanks a lot ,and I have figured it out. – lb lb Jul 01 '16 at 07:39 — lb lb, Jul 01 '16 at 07:39

Basile Starynkevitch · Answer 2 · 2016-06-24T09:02:03.413

3

You should read the Python tutorial.

You can create lists, dictionaries, objects and closures in Python. All these live in the (Python) heap, and Python has a naive garbage collector (reference counting + marking for circularity).

^{(the Python GC is naive because it does not use sophisticated GC techniques; hence it is slower than e.g. Ocaml or many JVM generational copying garbage collectors; read the GC handbook for more; however the Python GC is much more friendly to external C code)}

edited Jun 24 '16 at 09:02

answered Jun 24 '16 at 08:40

Basile Starynkevitch

223,805
18
296
547

Naive? As in credulous, innocenr, easily fooled? Lacking worldly experience? A bit judgemental, no? – rici Jun 24 '16 at 09:00

score 1 · Answer 3 · answered Aug 18 '16 at 08:47

1

Keep in mind that interpreted languages usually don't flatten the types as compiled languages do. The memory layout is (probably) completely different than in the raw data. Therefore, you cannot simply cast raw data to a class instance or vice versa. You have to read the raw data, interpret it and fill your objects manually.

answered Aug 18 '16 at 08:47

Viktor Engelmann

122
3

Technically, python is compiled. Interpreted languages are run character by character, whereas compiled languages are converted to bytecode and then are run by the appropriate Virtual Machine. – nerdguy Jun 17 '20 at 01:03

How to dynamically allocate memory in Python

3 Answers3