14

I've set up my code to carefully load and process data locally on my NUMA system. I think. That is, for debugging purposes I'd really like to be able to use the pointer addresses being accessed inside a particular function, which have been set up by many other functions, to directly identify the NUMA node(s) that the memory pointed at is residing on, so I can check that everything is located where it should be located. Is this possible?

I found this request on msdn http://social.msdn.microsoft.com/Forums/en-US/parallelcppnative/thread/37a02e17-e160-48d9-8625-871ff6b21f72 for the same thing, but the answer uses QueryWorkingSetEx() which appears to be Windows specific. Can this be done on Linux? I'm on Debian Squeeze, to be precise.

Thanks.

Rob_before_edits
  • 1,163
  • 9
  • 13

4 Answers4

24

There is an move_pages function in -lnuma: http://linux.die.net/man/2/move_pages

which can report current state of address(page) to node mappings:

nodes can also be NULL, in which case move_pages() does not move any pages but instead will return the node where each page currently resides, in the status array. Obtaining the status of each page may be necessary to determine pages that need to be moved.

So, call may be like:

 void * ptr_to_check = your_address;
 /*here you should align ptr_to_check to page boundary */
 int status[1];
 int ret_code;
 status[0]=-1;
 ret_code=move_pages(0 /*self memory */, 1, &ptr_to_check,
    NULL, status, 0);
 printf("Memory at %p is at %d node (retcode %d)\n", ptr_to_check, status[0], ret_code);
osgx
  • 90,338
  • 53
  • 357
  • 513
  • Using your answer I get "fatal error: numaif.h: No such file or directory". Do you have an idea what is wrong? – klm123 Nov 13 '13 at 06:58
  • 2
    Ok. I got it. The header is not included with glibc, but requires installing libnuma-devel or a similar package. – klm123 Nov 13 '13 at 07:01
  • @osgx I tried out your code. For some reason, status[0] is populated by `move_pages()` with a value of -2 even tho I tried to first initialize it to -2147483648. Any idea? The address I tried to pass it was a memory address returned by `mmap()` (with multiple 2MB hugepages) if it matters. Thanks! – HCSF Jan 12 '20 at 07:00
  • @HCSF, Could you open a new question with full example attached? Kernel version is required too. Then I will try to check this. – osgx Jan 13 '20 at 06:50
  • @osgx I updated my [post](https://stackoverflow.com/questions/54546367/fail-to-query-via-move-pages). Thanks! – HCSF Jan 13 '20 at 07:28
13

Alternatively, there is a get_mempolicy function in -lnuma: http://linux.die.net/man/2/get_mempolicy

If flags specifies both MPOL_F_NODE and MPOL_F_ADDR, get_mempolicy() will 
return the node ID of the node on which the address addr is allocated into 
the location pointed to by mode. If no page has yet been allocated for the 
specified address, get_mempolicy() will allocate a page as if the process had 
performed a read [load] access to that address, and return the ID of the node 
where that page was allocated.

Thus, the numa node of a page being pointed at by ptr is checked with:

int numa_node = -1;
get_mempolicy(&numa_node, NULL, 0, (void*)ptr, MPOL_F_NODE | MPOL_F_ADDR);
return numa_node;
mrlarkin
  • 131
  • 1
  • 2
  • Is there a reason both these functions (yours and accepted answer) perform the same behaviour? When should I use one other the other? – ffledgling Jun 26 '18 at 13:07
  • `get_mempolicy()` works for me (but `move_pages()` doesn't). Tho, it seems like `get_mempolicy()` cannot get the NUMA node IDs of multiple memory addresses in 1 call. Or am I missing something? – HCSF Jan 12 '20 at 07:14
0

If you know the address of your mmap return, try to check /proc/{pid}/numa_maps. You will see something like

14ab47200000 interleave:2-3 anon=1171400 dirty=1171400 N2=585690 N3=585710 kernelpagesize_kB=4

Which means 0x14ab47200000 address is interleaved to Node 2 and Node 3(N3). Check https://linux.die.net/man/5/numa_maps for details.

Popeye
  • 2,002
  • 20
  • 14
0

In kernel mode, we can use lookup_node().

Three Alley
  • 1
  • 1