Linux SCHED_FIFO not respecting thread priorities

Question

Scenario

I have created three threads, pinned to a single core, with the following priorities under SCHED_FIFO:

1. main: sched_priority = 99
2. thread_1: sched_priority = 97
3. thread_2: sched_priority = 98

The work threads (thread_1,thread_2) compute the sum of 50,000,000 primes (~ 10s). They do not block or perform system calls until the end (to print output).

The main thread sleeps for one second, and then checks the promises of the work threads to see if done.

---

Expected Behavior

The main thread is at the highest priority. According to sched:

A SCHED_FIFO thread runs until either it is blocked by an I/O request, it is preempted by a higher priority thread, or it calls sched_yield(2).

Main should therefore print (checking ...), in second intervals. It is highest priority so should preempt anything running. When it sleeps, it is blocking, so the other threads should run.

• thread_1: Finishes first, as it has priority when main is not busy.
• thread_2: Finishes last, and only starts after thread_1 is completely done.

---

Actual Behavior

The threads finish in the opposite order expected:

Thread 1 summed 3001134 primes at priority level: 97
Thread 2 summed 3001134 primes at priority level: 98
Main: Checking ...
Main: Task 1 has finished!
Main: Task 2 has finished!
Main: Exiting at priority level: 99

Reversing the priority orders so that main has the lowest yields the exact same result.

---

Reproduce

1. Compile program with g++ -o <exec_name> <file_name>.cpp -pthread
2. Run with: sudo taskset --cpu-list 1 ./<exec_name>

My kernel is 5.4.0-42-generic, and my distribution (if it matters): Ubuntu 18.04.5 LTS. I do not have the preempt-rt patch installed.

---

Similar Questions

• I have found this question that appears to describe the same problems, but no answer was given.

• I have also read in this question that my high priority thread can be preempted, but I don't care about this as long as it cannot be preempted by other threads born out of the same process. I do not have enough information about whether this could be happening.

---

Example Code

#include <thread>
#include <mutex>
#include <iostream>
#include <chrono>
#include <cstring>
#include <future>
#include <pthread.h>
#include <math.h>

// IO Access mutex
std::mutex g_mutex_io;

// Computation function (busy work)
static bool isPrime (unsigned int value)
{
    unsigned int i, root;
    if (value == 1)       return false;
    if (value == 2)       return true;
    if ((value % 2) == 0) return false;
    root = (int)(1.0 + sqrt(value));
    for (i = 3; (i < root) && (value % i != 0); i += 2);
    return (i < root ? false : true);
}

// Thread function
void foo (unsigned int id, unsigned int count)
{
    sched_param sch;
    int policy, sum = 0;

    // Get information about thread
    pthread_getschedparam(pthread_self(), &policy, &sch);

    // Compute primes
    for (unsigned int i = 1; i < count; ++i) {
        sum += (isPrime(i) ? 1 : 0);
    }

    // Print
    {
        std::lock_guard<std::mutex> lock(g_mutex_io);
        std::cout << "Thread " << id << " summed " << sum << " primes"
                  << " at priority level: " << sch.sched_priority << std::endl; 
    }

}

int main ()
{
    sched_param sch;
    int policy;

    // Declare and init task objects
    std::packaged_task<void(unsigned int, unsigned int)> task_1(foo);
    std::packaged_task<void(unsigned int, unsigned int)> task_2(foo);

    // Get the futures
    auto task_fut_1 = task_1.get_future();
    auto task_fut_2 = task_2.get_future();

    // Declare and init thread objects
    std::thread thread_1(std::move(task_1), 1, 50000000);
    std::thread thread_2(std::move(task_2), 2, 50000000);

    // Set first thread policy
    pthread_getschedparam(thread_1.native_handle(), &policy, &sch);
    sch.sched_priority = 97;
    if (pthread_setschedparam(thread_1.native_handle(), SCHED_FIFO, &sch)) {
        std::cerr << "pthread_setschedparam: " << std::strerror(errno) 
                  << std::endl;
        return -1;
    }

    // Set second thread policy
    pthread_getschedparam(thread_2.native_handle(), &policy, &sch);
    sch.sched_priority = 98;
    if (pthread_setschedparam(thread_2.native_handle(), SCHED_FIFO, &sch)) {
        std::cerr << "pthread_setschedparam: " << std::strerror(errno) 
                  << std::endl;
        return -1;
    }

    // Set main process thread priority
    pthread_getschedparam(pthread_self(), &policy, &sch);
    sch.sched_priority = 99;
    if (pthread_setschedparam(pthread_self(), SCHED_FIFO, &sch)) {
        std::cerr << "pthread_setschedparam: " << std::strerror(errno)
                  << std::endl;
        return -1;
    }

    // Detach these threads
    thread_1.detach(); thread_2.detach();

    // Check their status with a timeout
    for (int finished = 0; finished < 2; ) {
        std::this_thread::sleep_for(std::chrono::seconds(1));
        {
            std::lock_guard<std::mutex> lock(g_mutex_io);
            std::cout << "Main: Checking ..." << std::endl;
        }
        if (task_fut_1.wait_for(std::chrono::seconds(0)) == std::future_status::ready) {
            {
                std::lock_guard<std::mutex> lock(g_mutex_io);
                std::cout << "Main: Task 1 has finished!" << std::endl;
            }
            finished++;
        }
        if (task_fut_2.wait_for(std::chrono::seconds(0)) == std::future_status::ready) {
            {
                std::lock_guard<std::mutex> lock(g_mutex_io);
                std::cout << "Main: Task 2 has finished!" << std::endl;
            }
            finished++;
        }
    }
    pthread_getschedparam(pthread_self(), &policy, &sch);
    std::cout << "Main: Exiting at priority level: " << sch.sched_priority << std::endl;
    return 0;
}

Experiments

---

Running this program with two cores sudo taskset --cpu-list 1,2 results in the following bizarre output:

Thread 2 computed 3001134 primes at priority level: 98
Thread 1 computed 3001134 primes at priority level: 0
Main: Checking ...
Main: Task 1 has finished!
Main: Task 2 has finished!
Main: Exiting at priority level: 99

The priority of thread_1 is zero.

If I expand this to include three cores sudo taskset --cpu-list 1,2,3, then I get the behavior I expected I want on single-core:

Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Main: Checking ...
Thread 2 computed 3001134 primes at priority level: 98
Thread 1 computed 3001134 primes at priority level: 0
Main: Checking ...
Main: Task 1 has finished!
Main: Task 2 has finished!
Main: Exiting at priority level: 99

---

Rearranging the order in which the priorities are configured so that the main thread is done first, does not change output in the original scenario

Answer 1

When you start the two threads

// Declare and init thread objects
std::thread thread_1(std::move(task_1), 1, 50000000);
std::thread thread_2(std::move(task_2), 2, 50000000);

they may (!) immediately run and fetch the schedule parameters

// Get information about thread
pthread_getschedparam(pthread_self(), &policy, &sch);

even before you set them with pthread_setschedparam() to another value. The output might even show 0 and 0, if both threads are scheduled accordingly.

---

The child threads may (!) both be scheduled after the main thread has set the priority. Then you would get the expected output. But any result is possible.

---

When you move pthread_getschedparam() to the end of the thread just before the output, you are more likely to get the expected output of 97 and 98. But even then both threads may run until the end, even before the main thread is scheduled to set the priority.