Changing thread real time scheduling policy fails: CONFIG_RT_GROUP_SCHED=y

Question

My apologies if I'm posting this here instead of super user.

I was trying to run docker inside real-time group and I came across enabling cgroups - CONFIG_RT_GROUP_SCHED in the kernel to run real-time docker applications (here: https://docs.docker.com/config/containers/resource_constraints/#configure-the-default-cfs-scheduler)

I configured my kernel to enable the FIFO/RR flags and verified it (Available here: How to enable CONFIG_RT_GROUP_SCHED in Ubuntu to make it RT)

I believe my system is properly scheduled now because I'm able to run the resource limited docker on this system which accesses the cgroups with the following command:

$ docker run -it --cpu-rt-runtime=950000 \
                      --ulimit rtprio=99 \
                      --cap-add=sys_nice \
                      debian:jessie

I went ahead and tried to explore more features of RT system. I've this CPP code to assign RT priority scheduling to threads. This code basically tries to set the SCHED_FIFO priority to a thread and prints if the kernel allowed it to set the priority or not.

#include <iostream>
#include <pthread.h>
#include <sched.h>

using namespace std;

void set_realtime_priority() {
     int ret;
     // We'll operate on the currently running thread.
     pthread_t this_thread = pthread_self();
     // struct sched_param is used to store the scheduling priority
     struct sched_param params;

     // We'll set the priority to the maximum.
     params.sched_priority = sched_get_priority_max(SCHED_FIFO);
     std::cout << "Trying to set thread realtime prio = " << params.sched_priority << std::endl;

     // Attempt to set thread real-time priority to the SCHED_FIFO policy
     ret = pthread_setschedparam(this_thread, SCHED_FIFO, &params);
     if (ret != 0) {
         // Print the error
         std::cout << "Unsuccessful in setting thread realtime prio" << std::endl;
         return;     
     }
     // Now verify the change in thread priority
     int policy = 0;
     ret = pthread_getschedparam(this_thread, &policy, &params);
     if (ret != 0) {
         std::cout << "Couldn't retrieve real-time scheduling paramers" << std::endl;
         return;
     }

     // Check the correct policy was applied
     if(policy != SCHED_FIFO) {
         std::cout << "Scheduling is NOT SCHED_FIFO!" << std::endl;
     } else {
         std::cout << "SCHED_FIFO OK" << std::endl;
     }

     // Print thread scheduling priority
     std::cout << "Thread priority is " << params.sched_priority << std::endl; 
}

int main(){
set_realtime_priority();
return 0;
}

I've verified this code on a generic ubuntu/fedora and RT patched CentOS system. All of these systems allow the code to set the priority. Surprisingly its the CONFIG_RT_GROUP_SCHED=y configured kernel which doesn't allow me to set the priority policy. Similarly it also doesn't allow cyclictest to run

//install cyclictest by following

$ sudo apt-get install rt-tests
$ sudo cyclictest

I didn't understand this anomalous behavior. Does enabling the CONFIG_RT_GROUP_SCHED somehow block me from changing scheduling policies?

What is the value of `errno`? Which call does fail? `doesn't allow cyclictest to run` what do you mean by that? — KamilCuk, Jul 05 '19 at 14:27
@KamilCuk pthread_setschedparam() is unsuccessful because the kernel doesn't allow it to change scheduling policy. Cyclictest is a performance analysis tool which also tries to change the scheduling policy and the system responds with "Unable to change scheduling policy" — Abhay Nayak, Jul 08 '19 at 08:12

score 3 · Answer 1 · answered Mar 13 '20 at 05:50

I'm not sure you've got fixed it or not, but I met same problem yesterday and here is a solution of mine.

When CONFIG_RT_GROUP_SCHED=y, you should give a permission to run a(some) RT thread(s) to your cgroup. It can be done by giving proper value to "rt_runtime_us" node.

1) Find a cgroup for cyclictest

It follows the cgroup for shell process if it is lunched by shell command. Please enter below command to find the cgroup for cyclictest or sh process

] ps -O cgroup

2) Allocate proper timeslot for rt schedule to your cgroup

In my case, 'sh' process is into the "system.slice/system-serial-blabla" cgroup. So, I gave max timeslot to these cgroup hierarchy and it's been fixed.

] echo 950000 > /sys/fs/cgroup/cpu/system.slice/cpu.rt_runtime_us

] echo 950000 > /sys/fs/cgroup/cpu/system.slice/system-serial-blabla/cpu.rt_runtime_us

Good luck!

score 1 · Answer 2 · answered Jan 05 '22 at 18:56

As pointed out by Yongho Shin you still have to allocate a proper time slot for the control group where your cyclictest is run - irrespectively of whether you are running it from inside the Docker or outside of it - else it will show you the error message Unable to change scheduling policy! either run as root or join realtime group. As already pointed out by the error message itself you will have to run the process in the real-time cgroup as follows:

First let's install cgexec from the cgroup-tools:

$ sudo apt-get install cgroup-tools

Then allocate a time slice for the cgroup (in the following example 950000) that your process should run in by modifying the corresponding configuration file, e.g.:

$ sudo echo 950000 > /sys/fs/cgroup/cpu/system.slice/cpu.rt_runtime_us

Finally you can then launch a process inside the cgroup with something like

$ sudo cgexec -g subsystems:path_to_cgroup command arguments

In the case of our cyclictest our command could look like:

$ sudo cgexec -g cpu:system.slice cyclictest -m -sp99 -d0

Changing thread real time scheduling policy fails: CONFIG_RT_GROUP_SCHED=y

2 Answers2