6

I have an embedded linux system containing two threads that must run in real time (or soft real time). When using SCHED_OTHER, I noted a lot of jitter but the two threads always executed within their allocated time.

I have applied the RT patch with PREEMPT_RT enabled, and running those two threads with SCHED_FIFO (with a high thread priority of ~80) leads to a lot less jitter, it's overall a lot better, except once and a while both threads miss their deadline (instead of executing every 10 ms or so, they may not get schedule for almost a second!).

I wanted to ask which tool is best when debugging linux scheduling (under RT) on an embedded Linux OS. ftrace came to mind, but I don't know if it is the best and/or only tool. My goal is to find out why the two threads don't get scheduled for an extensive amount of time once in a while.

UPDATE: I've been running ftrace today with wakeup_rt. wakeup_rt as a tracer didn't get the job done: the max latency it recorded was 5ms when my thread can run up to 1000ms late. Maybe it is not a scheduler issue? What other tracer in ftrace would you recommend please?

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
Jary
  • 1,501
  • 3
  • 24
  • 35
  • What kind of CPU/platform are you using? Have you ruled out things like SMI on X86? – Frank Meerkötter Nov 06 '16 at 05:49
  • It is an ARM Cortex A9 CPU (I don't believe they have SMI). I am worried that a kernel thread (maybe an IRQ thread) is disabling preemption, maybe? I am not sure how to find out exactly though. – Jary Nov 06 '16 at 06:12
  • 1
    What you want is Ftrace. See http://stackoverflow.com/questions/22900469/debugging-linux-scheduler – gby Nov 08 '16 at 07:22
  • Check if this helps. see https://lwn.net/Articles/464296/ – Dheeraj Kumar Nov 08 '16 at 09:38
  • Thanks @gby. I've been running ftrace today with wakeup_rt. wakeup_rt as a tracer didn't get the job done: the max latency it recorded was 5ms when my thread can run up to 1000ms late. Maybe it is not a scheduler issue? What other tracer in ftrace would you recommend please? – Jary Nov 09 '16 at 07:18
  • [RT docs](https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO) mention changes in interrupt processing and `lpptest`, an interrupt latency benchmarking tool that comes with the patch, and some debugging capabilities built into it. – ivan_pozdeev Nov 14 '16 at 18:12

1 Answers1

0

Try using rt-app which is used by scheduler developers.

You might also want to use SCHED_DEADLINE instead of SCHED_FIFO to reduce your jitter.

idlethread
  • 1,111
  • 7
  • 16