Obtain load average on single CPU core on SMP system

Question

There is an API in Linux to get Load Average metric - system call getloadavg().

It obtains 1, 5 and 15 minutes Load Average, but seem that it's a common load for all the CPUs (all CPU cores). But how can I from regular C file obtain load average on single (specific) CPU core on SMP system?

You might get what you need from the per-CPU data in ```/proc/stat``` - [this thread](https://stackoverflow.com/questions/5514119/accurately-calculating-cpu-utilization-in-linux-using-proc-stat) might help — sj95126, Aug 10 '21 at 17:49
I don't think there's a per-CPU process queue. There's a single, system-wide process queue, and when a process is ready to run it's assigned to an appropriate CPU. The load average is the average length of that queue. — Barmar, Aug 10 '21 at 18:14
So there's no such thing as the load average on a specific CPU. — Barmar, Aug 10 '21 at 18:14
@Barmar Okay! But what about CPU affinity? Suppose I have one user-mode process which is tained to specific CPU core, how can I evaluate the overload of this core? — NK-cell, Aug 10 '21 at 18:19
You're not "overloading" that core. You're just increasing the overall load average because when that process is ready to run, the CPU might not be available, so it will stay in the run queue. — Barmar, Aug 10 '21 at 18:27
There might be statistics related to this, I suggest you ask on [unix.se]. — Barmar, Aug 10 '21 at 18:28

Andrew · Answer 1 · 2021-09-10T06:26:54.397

To evaluate average core load of an Intel CPU, you may read IA32_TIME_STAMP_COUNTER and IA32_MPERF MSRs.

Pseudocode:

for id in core_size
    read_msr(IA32_TIME_STAMP_COUNTER, TSC_1[id], id)
    read_msr(IA32_MPERF, MPERF_1[id], id)

do_work()

for id in core_size
    read_msr(IA32_TIME_STAMP_COUNTER, TSC_2[id], id)
    read_msr(IA32_MPERF, MPERF_2[id], id)
    tsc_d = TSC_2[id] - TSC_1[id]
    mperf_d = MPERF_2[id] - MPERF_1[id]
    if tsc_d < mperf_d
        print "core" id "load: 100.0"
    else
        print "core" id "load:" 100.0 - ((tsc_d - mperf_d) * 100.0 / tsc_d)

Edit - added C code (just for core 0):

// gcc avg_load.c -std=c11 -o avg
// gcc avg_load.c -std=c11 -lmsr -L$LIBMSR_PATH/lib/ -I$LIBMSR_PATH/include/ -DHAVE_MSR_CORE_H -o avg

#ifdef HAVE_MSR_CORE_H
    #include <msr_core.h>
#else
    #include <sys/types.h>
    #include <fcntl.h>
    #include <stdint.h>
#endif
#include <stdio.h>
#include <unistd.h>

#ifndef IA32_MPERF
    #define IA32_MPERF              0xE7
#endif

#ifndef IA32_TIME_STAMP_COUNTER
    #define IA32_TIME_STAMP_COUNTER 0x10
#endif

static int read_msr(int cpuid, off_t MSR_REGISTER_address, uint64_t *MSR_REGISTER_bits)
{
#ifdef HAVE_MSR_CORE_H
    return read_msr_by_idx(cpuid, MSR_REGISTER_address, MSR_REGISTER_bits);
#else
    char msr_file_name[64];
    sprintf(msr_file_name, "/dev/cpu/%d/msr_safe", cpuid);
    int fd = open(msr_file_name, O_RDONLY);
    if (fd < 0)
    {
        fprintf(stderr, "read msr error [%d]\n", cpuid);
        return -1;
    }

    if (pread(fd, MSR_REGISTER_bits, sizeof MSR_REGISTER_bits, MSR_REGISTER_address) != sizeof MSR_REGISTER_bits)
    {
        fprintf(stderr, "read msr error [%d]\n", cpuid);
        return -1;
    }
    close(fd);
    return 0;
#endif
}

int main()
{
#ifdef HAVE_MSR_CORE_H
    init_msr();
#endif
    uint64_t mperf_start, mperf_stop;
    uint64_t tsc_start, tsc_stop;

    read_msr(0, IA32_MPERF, &mperf_start);
    read_msr(0, IA32_TIME_STAMP_COUNTER, &tsc_start);
    
    sleep(1);
    
    read_msr(0, IA32_MPERF, &mperf_stop);
    read_msr(0, IA32_TIME_STAMP_COUNTER, &tsc_stop);
    
    uint64_t tsc_d   = tsc_stop - tsc_start;
    uint64_t mperf_d = mperf_stop - mperf_start;
    if (tsc_d < mperf_d)
        printf ("core 0 load: 100.0\n");
    else
        printf ("core 0 load: %f %\n", 100.0 - ((tsc_d - mperf_d) * 100.0 / tsc_d));

#ifdef HAVE_MSR_CORE_H
    finalize_msr();
#endif
    return 0;
}

The code expects to use msr_safe with the two used MSRs in the allowlist. Otherwise rewrite the code to read from /dev/cpu/msr with sudo rights.

Works without other dependencies or preferably with libmsr, but then requires compilation with -DHAVE_MSR_CORE_H flag.

According to my question and tags I'm trying to do it from C file. Is there any C API? — NK-cell, Sep 01 '21 at 10:32
Also IA32 things are not portable. I need my code to work not only on Intel CPUs — NK-cell, Sep 01 '21 at 10:33
msr_safe (https://github.com/LLNL/msr-safe) + libmsr (https://github.com/LLNL/libmsr) provides C API to read MSRs from user space a secure way. Using TSC and MPERF MSRs works for Intel and AMD. I am not aware about any sysfs interface or utility that would provide the load for every HW platform. — Andrew, Sep 02 '21 at 19:28
Just a note, msr_safe works for AMD CPUs fine, but libmsr not, however read/write functions can be easily implemented, see https://code.it4i.cz/vys0053/meric/-/blob/dev/src/basis/msrutil.h — Andrew, Sep 02 '21 at 22:08
I have `Intel(R) Core(TM) i7-6800K` CPU, `Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit`, but your bash script doesn't work with error: . Please provide complete executable example. ./1.sh: line 4: syntax error near unexpected token `read_msr' ./1.sh: line 4: ` read_msr(IA32_TIME_STAMP_COUNTER, TSC_1[id], id)' — NK-cell, Sep 03 '21 at 12:57
Sorry, for misunderstanding. This sample is a pseudocode. You can use `read_msr()` from the code I have referred to, or directly from libmsr. Do you need help with rewriting this sample to C? — Andrew, Sep 06 '21 at 05:27

Obtain load average on single CPU core on SMP system

1 Answers1