2

I am trying to list and sort all running processes in Windows by their memory consumption with the python psutil module. However when I query a processes memory_info attribute none of the various metrics will get above 4 gigabytes. Why is this limited and how can I get around this?

I did try using proc.memory_full_info() which would theoretically fix this issue with the uss memory metric, however I cannot figure out how to do this without causing an AccessDenied error.

sample script:

import psutil
from psutil._common import bytes2human

procs = [p.info for p in psutil.process_iter(attrs=['memory_info', 'memory_percent', 'name'])]

for proc in sorted(procs, key=lambda p: p['memory_percent'], reverse=True):
    print("{} - {}".format(proc['name'], bytes2human(getattr(proc['memory_info'], 'rss'))))

I'm open to any other way of profiling memory usage, psutil just seems to be the best tool I found out there.

Spencer
  • 1,931
  • 1
  • 21
  • 44
  • The question is: should it return something above *4GiB*? Also what does `sys.maxsize` return / print? – CristiFati Aug 15 '19 at 17:09
  • Yes, it definitely should.. Sorry I didn't include any specific data but I am currently running processes with 10G+ in taskmanager.exe and they still cap out at 4G. It terms of `sys.maxsize` do you think python just can't hold a number that big, if I understand you correctly? I'm not sure how that would work. – Spencer Aug 15 '19 at 17:18
  • To answer your question though, `sys.maxsize` returns `2147483647`. Instead of using the bytes2human function if I just return the actual byte size in psutil the maximum I currently get is `4294967295`. – Spencer Aug 15 '19 at 17:20
  • 2
    It's what I imagined. You're running *32bit* *Python*. Now I didn't get into *psutils* internals (the *C* code), but I assume it uses *32bit* unsigned integral types - which can hold values up to *4GiB*). Try installing a *64bit* *Python* and run your code there. – CristiFati Aug 15 '19 at 17:23
  • Well that's frustrating, but seems like you're right. Not loving the idea of switching my python version at this point but if that's what I have to do... Thanks for the help. – Spencer Aug 15 '19 at 17:27
  • You don't have to switch any versions. Just install the *64bit* version (for example I have more than 30 *Python* instances on my laptop), and run the code with that one. It should work perfectly (except some rare cases when code is extremely poorly written). – CristiFati Aug 15 '19 at 17:31
  • And updating all modules. My workstation isn't online so that is a manual process... I'm just surprised that psutil didn't get around that issue, it's just a query so I would have thought it isn't an issue. Ah well. – Spencer Aug 15 '19 at 17:45
  • Actually, I think it did for *64bit*. In general, *32bit* is now for backwards compatibility only. Also migrating the modules shouldn't be hard. `pip freeze` + `pip install -r`. – CristiFati Aug 15 '19 at 17:46
  • Yeah just finished and it's working great now, thanks again. – Spencer Aug 15 '19 at 17:56
  • You're welcome! I also added an answer with more explanations. – CristiFati Aug 15 '19 at 18:21

1 Answers1

0

You are using 32bit Python. By default a 32bit process has a 4GiB memory limit. On a software level, that limit comes from the underlying C types limits (and psutil's core is written in C).
For example, almost all [MS.Docs]: PROCESS_MEMORY_COUNTERS structure members have the SIZE_T type.

Example:

  • 32bit:

    >>> import sys, ctypes
>>> from ctypes import wintypes
>>> sys.version
'3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)]'
>>>
>>> ctypes.sizeof(wintypes.DWORD), ctypes.sizeof(ctypes.c_size_t), ctypes.sizeof(wintypes.HANDLE)
(4, 4, 4)

  • 64bit:

    >>> import sys, ctypes
>>> from ctypes import wintypes
>>> sys.version
'3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]'
>>>
>>> ctypes.sizeof(wintypes.DWORD), ctypes.sizeof(ctypes.c_size_t), ctypes.sizeof(wintypes.HANDLE)
(4, 8, 8)

For more details on how to distinguish between different CPU architectures Python processes, check: [SO]: How do I determine if my python shell is executing in 32bit or 64bit mode on OS X? (@CristiFati's answer)

CristiFati
  • 38,250
  • 9
  • 50
  • 87