1

Is there a Python package that can be used to find out if swap space was used during the execution of a function, and if it was used, the peak amount of usage? I know that psutil can be used to check the status quo of swap space as mentioned in this answer.

>>> import psutil
>>> psutil.swap_memory()
sswap(total=2097147904L, used=886620160L, free=1210527744L, percent=42.3, sin=1050411008, sout=1906720768)

However, I can't figure out the way to use measure the peak usage of swap space in a specific function. I found that below method can be used to measure the peak usage of memory for a specific function, but I think this only includes in-memory usage, not the swap space.

from memory_profiler import memory_usage
from time import sleep

def f():
    # a function that with growing
    # memory consumption
    a = [0] * 1000
    sleep(.1)
    b = a * 100
    sleep(.1)
    c = b * 100
    return a

mem_usage = memory_usage(f)
print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))

Is there any way to find out whether swap space was used, and if it was used, the peak swap space usage for a specific function in Python?

SHM
  • 61
  • 1
  • 8
  • 1
    I'm not convinced this is knowable (or meaningful). `psutil.swap_memory` return the system-wide swap usage, not process usage, because swap space isn't tracked on a per-process basis. You can know memory usage, but whether you have pages swapped or not is not predictable. – Tim Roberts Jan 01 '22 at 03:09
  • 1
    I agree with @TimRoberts here -- this is really hard to compute in general, highly OS, machine, and load-dependent, and really unlikely to be practically useful. What's the actual goal here? Are you concerned about or trying to reduce the memory usage of a Python program? Are you trying to learn or understand swap space on your machine? – bnaecker Jan 01 '22 at 04:09
  • To be more clear, there's no such thing as "the swap usage of a particular function". The OS (it sounds like we're talking about Linux here, but could be others) may decide to use swap space at some point, while a particular Python function is executing, but that's definitely not the same thing. – bnaecker Jan 01 '22 at 04:12
  • I am trying to find an optimal size for a cache that holds a specific data structure of a program that performs the same procedure multiple times. Let's say I have a total of 32GB of memory and set the cache size as 8GB. If the program's peak memory usage without total memory constraints is, say, 35GB (of course including 8GB cache usage), then in theory it will use 3GB of swap space under 32GB of memory constraint. – SHM Jan 01 '22 at 04:56
  • If I could measure this when I run a trial of one loop (before I actually fix the cache size), then I could decrement the cache size by 3GB (which is the amount of swap space usage) and fix it as 5GB when launching actual loops. Of course, the program will experience a slight performance degradation from decreased cache size, but this negative impact will be less than the gain coming from not using swap space at all. Thus I may assume that 5GB is the optimal cache size in this case. – SHM Jan 01 '22 at 04:56
  • 1
    This kind of optimization is almost hopeless. Virtual memory use is way too dynamic to be captured. You will get different results on different systems, and with different workloads on the computer. Your best plan is to pick a size (like your 5GB) and use it. The difference in performance between a 5GB cache and a 10GB cache is not going to measurable. – Tim Roberts Jan 01 '22 at 06:00

0 Answers0