How to find memory usage with memory_profiler Python?

Question

At the end of this question I found some code. I will put it here for convenience:

import memory_profiler as mp

def fun(n):
    tmp = []
    for i in range(n):
        tmp.extend(list(range(i*i)))
    return "XXXXX"

start_mem = mp.memory_usage(max_usage=True)
res = mp.memory_usage(proc=(fun, [100]), max_usage=True, retval=True) 
print('start mem', start_mem)
print('max mem', res[0][0])
print('used mem', res[0][0]-start_mem)
print('fun output', res[1])

But this doesn't work because res is not a double array, it is float number. Moreover, I don't understand how to check memory usage for many functions. I mean, would something like this work?

import memory_profiler as mp

def fun1(n):
    return "XXXXX"
def fun2(n):
    return "YYYYY"

methods = [
    'fun1(n)',
    'fun2(n)',
    ]

start_mem = mp.memory_usage(max_usage=True)
res = mp.memory_usage(proc=(methods[0], [100]), max_usage=True, retval=True) 
print('start mem', start_mem)
print('max mem', res[0][0])
print('used mem', res[0][0]-start_mem)
print('fun output', res[1])

score 3 · Answer 1 · edited May 27 '21 at 21:17

3

Installing memory_profiler:

pip3 install -U memory_profiler

Sometimes it is useful to have full memory usage reports as a function of time (not line-by-line) of external processes (be it Python scripts or not). In this case, the executable mprof might be useful

To get the line by line profiling, add decorator ‘@profile’ in front of the def function.

from memory_profiler import profile
@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

After this save the .py file and run the below code in terminal.

mprof run perf_test_v1.py

This will show you the memory usage by every line of code. To Plot the memory usage vs time through the below code.

mprof plot

The available commands for mprof are:

mprof run: running an executable, recording memory usage
mprof plot: plotting one the recorded memory usage (by default, the last one)
mprof list: listing all recorded memory usage files in a user-friendly way.
mprof clean: removing all recorded memory usage files.
mprof rm: removing specific recorded memory usage files

This would help you identify the bottlenecks in your codes and helps in optimizing the steps.

To get time-based memory usage

Execute the code passing the option -m memory_profiler to the python interpreter to load the memory_profiler module and print to stdout the line-by-line analysis. If the file name was example.py, this would result in:

$ python -m memory_profiler example.py

edited May 27 '21 at 21:17

halfer

19,824
17
99
186

answered May 22 '21 at 15:30

Pratap Alok Raj

1,098
10
19

Is it possible to use mprof without terminal? – slx0009 May 22 '21 at 15:39
There are two ways you can run it :- – Pratap Alok Raj May 22 '21 at 15:47
Sometimes it is useful to have full memory usage reports as a function of time (not line-by-line) of external processes (be it Python scripts or not). In this case, the executable mprof might be useful. – Pratap Alok Raj May 22 '21 at 15:48
Execute the code passing the option -m memory_profiler to the python interpreter to load the memory_profiler module and print to stdout the line-by-line analysis. If the file name was example.py, this would result in: $ python -m memory_profiler example.py – Pratap Alok Raj May 22 '21 at 15:48
Added in the answer itself, please check it once – Pratap Alok Raj May 22 '21 at 15:52
It is still terminal but I need something like this: `x = memory_profiler.memory_usage(fun1(10))` – slx0009 May 22 '21 at 16:13
There are other libraries which can help then like Guppy, Pympler etc.Try those they are also good – Pratap Alok Raj May 22 '21 at 16:30

score 0 · Accepted Answer · answered Feb 10 '23 at 07:15

I'll try to complement Alok's answer.

memory_usage method takes tuple (method, args, kwargs).

To find memory usage of a method and store value in variable, you can do this:

from memory_profiler import memory_usage


def foo(n):
    t = [list(range(i * i)) for i in range(n)]


if __name__ == '__main__':
    memory = memory_usage((foo, (5,), {}))
    print(memory)

If you have multiple methods, you can put them in list and then iterate over it:

from memory_profiler import memory_usage


def foo(n):
    t = [list(range(i * i)) for i in range(n)]


def bar(n):
    t = [1 for i in range(n)]


if __name__ == '__main__':
    methods = [foo, bar]
    memory = memory_usage((methods[0], (5,), {}))

Don't forget about using memory_usage in if __name__ == '__main__': or in other methods and don't forget about comma in (5,). This code will throw two exceptions because of __main__ and comma:

from memory_profiler import memory_usage


def foo(n):
    t = [list(range(i * i)) for i in range(n)]


def bar(n):
    t = [1 for i in range(n)]


functions = [foo, bar]
memory = memory_usage((functions[0], (5), {}))

Full information about memory_usage you can find in documentation.

How to find memory usage with memory_profiler Python?

2 Answers2