Detect memory swapping in Python

Question

How to detect when OS starts swapping some resources of the running process to disk?

I came here from basically the same question. The psutil library is obviously great and provides a lot of information, yet, I don't know how to use it to solve the problem.

I created a simple test script.

import psutil
import os
import numpy as np

MAX = 45000

ar = []
pr_mem = []
swap_mem = []
virt_mem = []

process = psutil.Process()

for i in range(MAX):
    ar.append(np.zeros(100_000))
    pr_mem.append(process.memory_info())
    swap_mem.append(psutil.swap_memory())
    virt_mem.append(psutil.virtual_memory())

Then, I plotted the course of those statistics.

import matplotlib.pyplot as plt

plt.figure(figsize=(16,12))

plt.plot([x.rss for x in pr_mem], label='Resident Set Size')
plt.plot([x.vms for x in pr_mem], label='Virtual Memory Size')
plt.plot([x.available for x in virt_mem], label='Available Memory')
plt.plot([x.total for x in swap_mem], label='Total Swap Memory')
plt.plot([x.used for x in swap_mem], label='Used Swap Memory')
plt.plot([x.free for x in swap_mem], label='Free Swap Memory')

plt.legend(loc='best')
plt.show()

I cannot see, how I can use the information about swap memory to detect the swapping of my process.

The 'Used Swap Memory' information is kind of meaningless, as it is high from the very first moment (it counts global consumption) when the data of the process are obviously not swapped.
It seems best to look at the difference between 'Virtual Memory Size' and 'Resident Set Size' of the process. If VMS greatly exceeds RSS, it is a sign that the data are not on RAM but on disk.
- However, here is described a problem with sudden explosion of VMS which makes it irrelevant in some cases, if I understand it correctly.
Other approach is to watch 'Available Memory' and be sure that it does not drop below a certain threshold, as psutil documentation suggests. But to me it seems complicated to set the threshold properly. They suggests (in the small snippet) 100 MB, but on my machine it is something like 500 MB.

So the question stands. How to detect when OS starts swapping some resources of the running process to disk?

I work on Windows, but the solution needs to be cross-platform (or at least as cross-platform as possible).

Any suggestion, advice or useful link is welcomed. Thank you!

For the context, I write a library which needs to manage its memory consumption. I believe that with the knowledge of the program logic, my manual swapping (serializing data to disk) can work better (faster) than OS swapping. More importantly, the OS swap space it limited, thus, sometimes it is necessary to do the manual swapping which does not utilize the OS swap space.

In order to start the manual swapping at the right time, it is crucial to know when OS swapping starts.

Detect memory swapping in Python

0 Answers0