0

I'm currently working on an image processing script which is eating up a lot of memory, and I'm curious about which parts of the script are using up the largest part of the memory, so that I can optimise those parts in order to save on total memory usage (and therefore being able to process larger images).

However, there are a lot of different clauses and different processing functions being called within my script, which makes it hard to keep track of all of the variables that are currently allocated some (or a lot of) memory. I've already made it so several functions get called as a newly spawned process (using multiprocessing), in order to ensure they don't leave behind any of their variables after running. However, I'm still seeing a lot of memory being used in some situations, and I'd like to be able to pin-point which variables are taking up the largest amount of space.

I hope it's possible to somehow get a report of the used memory. I wouldn't mind having to install some additional packages to make this possible.

Joeytje50
  • 18,636
  • 15
  • 63
  • 95
  • 1
    Some ideas here: https://stackoverflow.com/questions/552744/how-do-i-profile-memory-usage-in-python. For general informations, you could use cProfile too, with, for example `python -m cProfile -s cumtime myscript.py` – Rivers Feb 08 '21 at 12:03
  • How would cProfile work to profile an entire script? Would I call my main function using `cProfile.run('main()')` where I currently simply call `main()`, for example? – Joeytje50 Feb 08 '21 at 12:07
  • 1
    No, in commmand line, just call python as I wrote. Open command line and type `python -m cProfile -s cumtime myscript.py`. And here you will find a function to know the size of a specific object in memory: https://stackoverflow.com/questions/449560/how-do-i-determine-the-size-of-an-object-in-python – Rivers Feb 08 '21 at 12:08
  • Not sure, but this way it looks like it crashes at the first instance of me spawning another process using `multiprocessing.Process` and then `.start()`ing that process. Does cProfile not support profiling a program using multiple processes? – Joeytje50 Feb 08 '21 at 12:21
  • cProfile with multiprocessing: https://stackoverflow.com/questions/11041683/python-multiprocess-profiling – Rivers Feb 08 '21 at 12:23

1 Answers1

1

I use module "memory_profiler".

pip install memory_profiler

from memory_profiler import profiler

@profiler
def xxx():
  print("Hello World!")
Himpq
  • 36
  • 3
  • But this module can only view the occupation of a certain line of code – Himpq Feb 08 '21 at 13:44
  • This actually looks very useful. However, you say it only views the memory usage of a certain line? Going by https://pypi.org/project/memory-profiler/ it shows the full memory usage of the entire function, right? What did you mean with your comment? – Joeytje50 Feb 08 '21 at 13:46
  • You can view the memory usage of each line of the function, but you cannot view the memory usage of variables with "@profiler". Baidu translation is not very accurate, the expression may be wrong. My English is poor. – Himpq Feb 08 '21 at 13:51
  • This looks like it's probably very useful! Thanks, I'll check it out, and if this is what I needed I'll mark your answer as accepted! – Joeytje50 Feb 08 '21 at 14:09