2

I have a server with 2 CPU's and 64GB of RAM, 32GB per CPU.

I know that each CPU has it's own part of RAM, lets call them RAM1 and RAM2. I would like to make my program know on which RAM (RAM1 or RAM2) it allocates it's data.

I tried to check pointers values:

  // put the thread at i-th CPU, using pthread_setaffinity_np
TData *a = new TData[N];
...
cout << "CPU = " << i << " adress = " << a << endl;

but the output looks random. I suppose that is because addresses are virtual. Is there any correspondence between virtual memory addresses and part of the RAM?

How to check in which RAM my array "a" is allocated?

klm123
  • 12,105
  • 14
  • 57
  • 95
  • possible duplicate of [How to allocate memory in a specific place in memory (C programming)?](http://stackoverflow.com/questions/17524067/how-to-allocate-memory-in-a-specific-place-in-memory-c-programming) – perreal Nov 06 '13 at 13:59
  • The possible duplicate looks relevant and might solve @klm123's problem. However, the questions are different. This question is about how to check after the fact where the memory is allocated, that question is about dictating up front where the memory should be allocated. – Magnus Hoff Nov 06 '13 at 14:06
  • @perreal, I saw this question and answer to it. How does it answer on my question? – klm123 Nov 06 '13 at 14:15
  • @klm123, as far as I understand from the answer, what you are trying to do is already the default behaviour. But if you really want to verify a local allocation then sorry for the close vote. – perreal Nov 06 '13 at 14:19
  • @perreal, Yes, I want to know a way to verify this, because it looks like I have a problems with memory allocation. My CPU's are randomly either fully or partially loaded. May be the reason is different, by I want to be sure. – klm123 Nov 06 '13 at 14:21
  • possible duplicate of [Can I get the NUMA node from a pointer address (in C on Linux)?](http://stackoverflow.com/questions/7986903/can-i-get-the-numa-node-from-a-pointer-address-in-c-on-linux) – Hristo Iliev Nov 12 '13 at 16:39

2 Answers2

9

Your question is answered here. I would only like to add some comments.

Note that calling new [] does not actually allocate physical memory. On modern operating systems this only results in an anonymous memory mapping begin made. Anonymous mappings do not correspond to files in the file system but rather are backed by the swap (if any). Initially the whole area points to a read-only page in the kernel that contains all zeros. Only when you actually write to the newly allocated memory, a new memory page is installed, which replaces the zero page for the page range where the accessed address falls. This is why we say that the zero page is copy-on-write (or CoW) mapped to the virtual address space of the process. The default policy is to try to allocate the new page on the same NUMA node, where the thread that accessed the memory area runs. This is called a "first-touch" NUMA policy. If there is not enough memory on that NUMA node, the page is allocated on some other node with enough free memory. It is also possible for small allocations to end up inside a bigger area (called arena), managed by the C library memory allocator malloc() (the C++ operator new [] calls malloc() in order to do the actual memory allocation). In this case the pages might be already present in the physical memory even before you write to the newly allocated memory.

Linux has the nasty habit of not preserving the NUMA association of memory areas when it swaps. That is, if a page was allocated on NUMA node 0, then swapped out and then swapped back in, there is no guarantee that the page won't be placed on NUMA node 1. This makes the question "where is my memory allocated" a bit tricky, since a consecutive swap-out followed by a swap-in could easily invalidate the result that you get from move_pages() just a fraction of a second ago. Therefore the question only makes sense in the following two special cases:

  • you explicitly lock the memory region: one could use the mlock(2) system call in order to tell the OS not to swap a particular range from the process virtual address space;
  • your system has no active swap areas: this prevents the OS altogether from moving the pages out from and back into the main memory.
Community
  • 1
  • 1
Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
  • Thanks! Using your link I get "fatal error: numaif.h: No such file or directory". Do you have an idea what is wrong? – klm123 Nov 13 '13 at 06:58
  • Ok. I got it. The header is not included with glibc, but requires installing libnuma-devel or a similar package. – klm123 Nov 13 '13 at 07:06
2

Memory is virtualized through the MMU, so each process sees a memory space of size equal to 2^64. Within the process, addresses are virtual, so they are meaningless. There isn't any corrispondence between the virtual addresses (seen by the application) and the physical addresses (on RAM) at the process-level.

Your application should query the operating system to know which physical addresses it is currenyly using.

Claudio
  • 10,614
  • 4
  • 31
  • 71
  • Given the PID of your application (that you can get by running *getpid()* in the C++ code), there should be some entry in */proc/PID/*. It could be */proc/PID/map*. – Claudio Nov 06 '13 at 14:05
  • my application is run on both cpu's. – klm123 Nov 06 '13 at 14:14
  • `/proc/PID/maps` does not expose any NUMA-related information. `/proc/PID/numa_maps` is the one that exposes but it only tells for each mapping how many pages are allocated on each NUMA node. One cannot get the required information from `/proc`. – Hristo Iliev Nov 12 '13 at 17:03
  • Just to make it clear: if the array is big enough, it will get its own mapping, and then one could parse `/proc/self/numa_maps`, find the mapping and find out how many pages are allocated on each NUMA node. If the array is embedded in a pre-existing mapping, then one cannot tell how many of the pages listed in `/proc/PID/numa_maps` relate to the subrange occupied by the array. Another problem is that even new mappings are sometimes coalesced with pre-existing ones and thus the second situation is realised. – Hristo Iliev Nov 12 '13 at 17:20