4

I'm working on a NUMA architecture, where each compute node has 2 sockets and 4 cores by socket, for a total of 8 cores by compute node, and 24GB of RAM by node. I have to proof that setting processor affinity can have a significant impact on performances.

Do you have any program to suggest that I could use as a benchmark to show the difference of impact between using processor affinity or not? I could also write a simple C test program, using MPI, or OpenMP, or pthreads, but what operation would be the best to do that test? It must be something that would take advantage of cache locality, but that also would trigger context switching (blocking operations) so process could potentially migrate to another core, or worse, to an other socket. It must run on a multiple of 8 cores.

Charles Brunet
  • 21,797
  • 24
  • 83
  • 124

3 Answers3

3

I tried to write a program that benchmarks asymmetry in memory latency on NUMA architecture, and with the help of the StackOverflow community, I succeeded. You can get the program from my StackOverflow post.

Measuring NUMA (Non-Uniform Memory Access). No observable asymmetry. Why?

When I run my benchmark program on hardware very similar to yours, I see about a 30% performance penalty when a core is reading/writing to memory that is not in the core's NUMA node (region of affinity). The program has to read and write in a pattern that deliberately defeats caching and pre-fetching, otherwise there's no observable asymmetry.

Community
  • 1
  • 1
James Brock
  • 3,236
  • 1
  • 28
  • 33
  • +1 on that. Given the specs you have (2 sockets, 24 GB) it sounds like you have the same type of system as the one in that question. – Mysticial Sep 02 '11 at 13:45
2

Try ASC Sequoia benchmark -- CLOMP -- designed for measuring threading overheads.

Avadhut Sardeshmukh
  • 21
  • 1
0

You can just use a simple single-threaded process which writes and then repeatedly reads a modest data set. The process needs to run for a lot longer than a single time slice, obviously, and long enough for processes to migrate from one core to another, e.g. 100 seconds.

You can then run two test cases:

  1. run 8 instances of the process without CPU affinity

    $ for p in 0 1 2 3 4 5 6 7 ; do time ./my_process & ; done

  2. run 8 instances of the process with CPU affinity

    $ for p in 0 1 2 3 4 5 6 7 ; do time taskset -c $p ./my_process & ; done

Paul R
  • 208,748
  • 37
  • 389
  • 560
  • But in that case, what would cause processes to migrate to another core? There are reasonable chances that if OS takes control of one core, then process would continue its execution of the same core when it regains control, even without setting affinity. – Charles Brunet Apr 08 '11 at 14:12
  • @Charles: this is just based on empirical observations - I've done simlar tests and have seen significant differences with/without CPU affinity. I can only guess that the various background processs running on a typical system are responsible for triggering migration from one core to another. – Paul R Apr 08 '11 at 14:46