6

The Windows API offers the CreateFileMappingNuma function (http://msdn.microsoft.com/en-us/library/windows/desktop/aa366539(v=vs.85).aspx) to create a named shared memory space on a specific NUMA node.

So far, I have not found an equivalent function for Linux.

My current approach looks like this:

  1. Allocate named shared memory (using shm_open(...))
  2. Determine current NUMA node (using numa_move_pages(...))
  3. Move pages to target Node (using numa_move_pages(...) again)

Does anyone know a better approach?

EDIT: For the record: My proposed implementation does work as expected!

Ben
  • 203
  • 1
  • 2
  • 8
  • Note that other apps mapping the same memory may move it to a different node later. On top of that, it is always better to have memory on "consumer" node as writes to a remote memory do not usually stall compared to reading from a remote memory. –  Mar 05 '14 at 18:03
  • I moved the memory to a specific node for that exact reason (to get the data close to a CPU). However, the scheduler is quite good at doing that himself, as I found out ;-). The only advantage this offers is, that all the memory pages are already in the right place, so you don't have the "priming" phase in the beginning of the program. – Ben Mar 06 '14 at 14:30

1 Answers1

0

That sounds right. Note that there are no pages allocated at the point where you call shm_open()/fruncate() (don't forget ftruncate() to set the size!). The kernel simply creates the vma and waits for future code accesses to fault the pages into physical memory. So calling numa_move_pages() in this state will presumably have the effect of creating and populating new pages in the relevant NUMA nodes.

Andy Ross
  • 11,699
  • 1
  • 34
  • 31