Pthread Priority and Scheduling

Question

I am trying to use Pthreads to set priorities and scheduling policies.

I modified a simple socket (downloadable online), with a server and several clients. These clients are simply sending a string to the server.

The server is creating a new thread for each client connecting and is setting a higher priority to every new connection, which means the last client connected is the one with the highest priority. The result I am expecting is that the thread with highest priority is the one printing out until that client is connected, while the others should wait.

However, this is not the result I am getting. The threads are all executed anyway, even the ones with lower priorities. I even tried several configurations, for instance using pthread_join, but in this case the scheduler will wait until that thread has finished its execution, which is wrong if another thread with higher priority is called (in my example another client connecting), since it should release the CPU to this newly created thread.

Here is the code:

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <pthread.h>
#include <sched.h>
#include <sys/time.h>

#define PORTNUMBER 8888

int prioInit = 30; //global variable to initialize priority

struct serverParm 
{
       int connectionDesc;
};

//function used to increase priority to new threads
void addPrio(int *prio)
{
  *prio = *prio + 10;
}

void *serverThread(void *parmPtr) 
{
  #define PARMPTR ((struct serverParm *) parmPtr)
  int recievedMsgLen;
  char messageBuf[1025];

  struct sched_param Priority_Param; //struct to set priority
  int policy=SCHED_FIFO; //kind of policy desired, either SCHED_FIFO or SCHED_RR, otherwise Linux uses SCHED_OTHER

  // Server thread code to deal with message processing
  printf("DEBUG: connection made, connectionDesc=%d\n",
        PARMPTR->connectionDesc);

  pthread_t self_id= pthread_self(); // I ask for the tid..
  Priority_Param.sched_priority = prioInit; //.. set the priority (the higher the sooner it is executed, min at 1, max at 99)..
  addPrio(&prioInit); //..increase the base priority..
  if(pthread_setschedparam(self_id, policy, &Priority_Param) != 0) //.. and set the scheduling options.
  {
    printf("Error Set Sched\n");
    perror("");
    exit(1);
  }

  if (PARMPTR->connectionDesc < 0) 
  {
    printf("Accept failed\n");
    exit(1);   
  }

  // Receive messages from sender
  while ((recievedMsgLen=
        read(PARMPTR->connectionDesc,messageBuf,sizeof(messageBuf)-1)) > 0) 
  {
    recievedMsgLen[messageBuf] = '\0';
    printf("Message: %s\n",messageBuf);
  }
  close(PARMPTR->connectionDesc);  // Avoid descriptor leaks 
  free(PARMPTR);                   // And memory leaks                        
}

int main () 
{
  int listenDesc;
  struct sockaddr_in myAddr;
  struct serverParm *parmPtr;
  int connectionDesc;
  pthread_t threadID;

  // Create socket from which to read 
  if ((listenDesc = socket(AF_INET, SOCK_STREAM, 0)) < 0) 
  {
    perror("open error on socket");
    exit(1);
  }

  // Create "name" of socket
  myAddr.sin_family = AF_INET;
  myAddr.sin_addr.s_addr = INADDR_ANY;
  myAddr.sin_port = htons(PORTNUMBER);

  if (bind(listenDesc, (struct sockaddr *) &myAddr, sizeof(myAddr)) < 0) 
  {
    perror("bind error");
    exit(1);
  }

  // Start accepting connections
  listen(listenDesc,5);

  while (1)
  {
    // Wait for a client connection
    connectionDesc = accept(listenDesc, NULL, NULL);

    // Create a thread to actually handle this client
    parmPtr = (struct serverParm *)malloc(sizeof(struct serverParm));
    parmPtr->connectionDesc = connectionDesc;
    if (pthread_create(&threadID, NULL, serverThread, (void *)parmPtr) 
          != 0) 
    {
      perror("Thread create error");
      close(connectionDesc);
      close(listenDesc);
      exit(1);
    }

    printf("Parent ready for another connection\n");
  }

}

I am compiling using -pthread option and running as root, so that the policy and priority is set and changed. I am for sure missing something here, but I am also wondering if it is possible to really use and change the scheduling options, in order to have a behavior similar to the real-time one.

Answer 1

Scheduler policy and priority is used to determine which thread will run when a choice must be made between two or more runnable threads.

If your high priority thread blocks in read(), then it is not eligible to run, and a lower priority thread that is runnable will get a chance to run instead. What the priority means is that even all of the available CPU cores are running lower priority threads at the moment when data arrives from the network for the higher priority thread, a lower priority thread will be immediately pre-empted so that the higher priority thread can run.

If you want to see the effect of priority, have a lower-priority thread do lots of work so that it is almost always using the CPU: your higher-priority thread should still respond immediately to network input.

(But if you do this, either use a non-realtime priority level for the lower-priority threads, or make sure you have a shell open that is running at an elevated realtime priority, so that the lower-priority thread doesn't block your shell from running when you need it).

Side note: your addPrio() function is racy - it should lock a mutex around the changes to *prio.

Answer 2

Regarding your ultimate goal of evaluating Linux's real behaviour in comparison to one with the premp-rt patch, the answer is not difficult.

Of course it behaves similarly, it's just that the latencies are higher are wildly variable, but still bounded.

You can write real time software for stock Linux that will work perfectly satisfactorily, provided that your real time response requirements aren't that challenging. If those requirements are to respond within a few milliseconds then you are going to need the premp-rt patches. If your requirements are tighter even than that then you'd need a different OS (VxWorks, etc).

There's some really good advise here and especially here. Linux preempt-rt and real time is not perfect, especially on Intel hardware, but if your requirements aren't that tight then you can do very well.

I can highly recommend downloading and applying preempt-rt yourself. That way you can compile FTRACE into your kernel and then you can play with kernelshark. You can put FTRACE Into a stock kernel too. Kernelshark is extremely useful when developing multi threaded applications, especially when you turn on task plots.