How to get 64-bit value from WMI in python

Question

When I run the following code:

def get_process_info(pid):
    c = wmi.WMI(namespace='root\\cimv2')
    obj = c.Win32_Process(ProcessId = pid)[0]
    print "VirtualSize:", obj.wmi_property('VirtualSize').type
    print "VirtualSize:", obj.wmi_property('VirtualSize').Value

def get_perf_info(pid):
    c = wmi.WMI(namespace='root\\cimv2')
    obj = c.Win32_PerfFormattedData_PerfProc_Process(IDProcess = pid)[0]
    print "PrivateBytes:", obj.wmi_property('PrivateBytes').type
    print "PrivateBytes:", obj.wmi_property('PrivateBytes').Value

Against a process which is using a lot of memory I get this:

VirtualSize: uint64
VirtualSize: 5015498752
PrivateBytes: uint64
PrivateBytes: 4294967295

Note that both are listed as being 64-bit values but the PrivateBytes values is 0xFFFFFFFF. If I use "WMI Explorer" I can see the PrivateBytes value is larger than 32-bits:

My question is how can I access PrivateBytes in its full 64-bit glory?

Is there a completely other way to read the WMI from python besides this WMI module?

You might find this of interest: http://stackoverflow.com/questions/1984186/what-is-private-bytes-virtual-bytes-working-set — Jacques Gaudin, Oct 08 '16 at 11:30
@JacquesGaudin. Thanks yes interesting. This isn't for debugging leaks but just kind of a estimate, so I think private bytes will do. I just want to know is it using 2GB or 6GB or 22GB just to get some idea. BTW I tried using psutil library and it also was mysteriously bound to 32-bit value. This is with 32-bit Python but that shouldn't matter, I believe some other values are 64-bit (python ints being any size). — Philip, Oct 09 '16 at 01:56
It may actually matter what version of Python/PyWin32 you're using. Not sure if it's applicable, but I found [this reference](https://msdn.microsoft.com/en-us/library/aa393067(v=vs.85).aspx) on requesting WMI data. Sounds like 32-bit applications requesting WMI data will be returned by 32-bit providers defaultly. Would you be willing to test if this works on 64-bit Python? — sytech, Oct 09 '16 at 07:02
FYI, I could not reproduce the issue with Python 2.7.11 64-bits. Have you tried `long(obj.wmi_property('PrivateBytes').Value)` ? — Jacques Gaudin, Oct 10 '16 at 11:38
Cast does not work. Must be 32-bit internally. What's frustrating is my 32-bit Python fetches a 64-bit VirtualSize value, but PrivateBytes is only 32-bit . With 64-bit Python both are 64-bit. I wonder if it's a bug in the WMI package, or in routines that it calls. — Philip, Oct 10 '16 at 15:19
The bug must come from `win32com.client`. I had a look at the `wmi` source and it is just a wrapper. There is nothing that leads to think it treats `VirtualSize` and `PrivateBytes` differently. — Jacques Gaudin, Oct 10 '16 at 23:40

score 2 · Accepted Answer · answered Oct 09 '16 at 19:24

2

You can use wmic which provides a command-line interface for WMI :

def find_privatebytes(pid):
    with os.popen('wmic process list full /format:csv') as csvfile:
        reader = csv.reader(csvfile, delimiter=',', quotechar='"')
        crow = 0
        col_pid = 0
        col_pb  = 0

        for row in reader:
            if len(row) == 0:
                continue
            if crow == 0:                
                col_pid = row.index("ProcessId")
                col_pb  = row.index("PrivatePageCount")
                crow += 1
            elif int(row[col_pid]) == pid:
                return int(row[col_pb])
        return 0

answered Oct 09 '16 at 19:24

napuzba

6,033
3
21
32

Seems like a good idea. But when I run it I get the error "ValueError: invalid literal for int() with base 10: 'Microsoft Windows 10 Pro|C: \\WINDOWS|\\Device\\Harddisk0\\Partition2'". And when I dump the output of "wmic process list full /format:csv" into Excel I see the columns are not lined up. It seems like some very long process names (chrome?) include commas and this messes up the format. Not sure if there is another /format which is more robust? – Philip Oct 10 '16 at 15:40
You can try using `/format:textvaluelist.xsl` which give `name=value` in a separate lines – napuzba Oct 10 '16 at 16:18
Found a better wmi query "wmic path win32_perfformatteddata_perfproc_process where "IDProcess=%d" get PrivateBytes /format:textvaluelist'. But it *still* doesn't work from 32-bit python. Somehow even wimc knows it's a 32-bit process! Doesn't seem possible. – Philip Oct 11 '16 at 02:15
What you get when you invoke the command from command line? what you get when invoking from python? – napuzba Oct 11 '16 at 06:53
Amazingly to me 32-bit python gives 0xFFFFFFFF, 64-bit python gives 64-bit value, and command line gives 64-value. I don't see how that's possible unless Python opens a 32-bit command process of some kind. [See gist](https://gist.github.com/anonymous/eaba32025198d3c9bcfe3bf7eb2a0148) – Philip Oct 11 '16 at 13:23
I'm thinking maybe I need a way that doesn't use wmi at all. I thought psutil would do it but it produces the same problem. – Philip Oct 11 '16 at 13:28
It seems that python is invoking the 32 bit version of `wmic`. Try to invoke by full path `C:\Windows\SysWOW64\wbem\wmic.exe` or `C:\Windows\System32\wbem\wmic.exe`. one of the programs is the 64 bit version. – napuzba Oct 12 '16 at 18:48
That doesn't work either! Here is the reason I believe: [microsoft post](https://blogs.msdn.microsoft.com/dsadsi/2009/09/18/accessing-a-64-bit-wmi-provider-from-a-32-bit-application-running-on-a-64-bit-client/). You need to set the special __ProviderArchitecture and __RequiredArchitecture to be 64-bit. I cannot figure out how to do this. Anything should be possible via Com but it's not easy. I think for us just waiting until we upgrading to Python 64-bit is a reasonable solution. Since this was 100X more complicated than hoped. – Philip Oct 13 '16 at 03:14
To make it more clear, once I read about ProviderArchitecture flag and didn't see that wimc had that option, I went back to the WMI python module. But could not figure out how to request 64 bit there. I did find an example of pure-com (no WMI module) which read values using only like 3 lines. So it's not exactly clear one needs WMI module. But again couldn't make that work. – Philip Oct 13 '16 at 03:23
Using os.stat on the above paths, It seems python refer the same file which is the 32 bit version of wmic. Try to copy 'C:\Windows\System32\wbem\wmic.exe' (64 bit version) to python script directory and then invoke the copied program. – napuzba Oct 13 '16 at 06:08
This worked! I modified the answer with the version that actually did PrivateBytes (not PrivatePageCount). Awarded the bounty. However a version that used WMI would still be better. There is a 5+ second delay the first time this is called, and copying the wmic.exe is kind of ugly. It should be possible to do this with WMI if you can pass ProviderArchitecture to request 64-bit but I could't figure it out. Thanks, great persistence! – Philip Oct 13 '16 at 13:28

How to get 64-bit value from WMI in python

1 Answers1