1

In experimenting with scheduling a high speed thread, I've noticed that sometimes there are large periods (multiple ms) that the thread is not scheduled. I was wondering what might be able to do this to the scheduler in the configuration I am using.

  • Latest Debian stretch (stable) with RT (i686)
  • Dual core Intel (2237MHz)
  • PS2 keyboard and mouse
  • CPU 1 reserved (Via grub)
  • Taskset used to put my test process onto core 1
  • SMI disabled, speed step disabled, etc (Typical RT BIOS settings)
  • USB ports not disabled but nothing plugged in
  • IRQ balance disabled

I confirmed that only things like worker, timer, etc are on CPU1. Everything else is on CPU0.

My thread is SCHED_DEADLINE (The only one scheduled like that) with a period of 300us. I use schedules_yield() to release the thread long before the reservation is up (Just a test loop) so I'm sure it's not over running.

What I get is an almost perfect 3333Hz output (Via an o-scope) which gets blocked every now and then for upward of 15ms at a time. I've looked at interrupts (Which aren't incrementing on that CPU except for the timer), I've disabled NMI interrupts and so on with no luck in finding what the interfering process is. I don't believe I fully understand what can take priority and cause the scheduler to skip periods so I'm hoping someone might have an idea?

I thought it might be disk IO, but that doesn't seem to align with the gaps (Sometimes it does...). VGA/console use seems to make it worse, but even when not being used the gaps still appear.

And yes, before you ask.... This is just an experiment to see if this can be done reliably. My actual code runs on QNX which is rock-steady at this rate on the same hardware. I'm experimenting to see if that can be ported to Debian with PREEMPT_RT.

Thanks!

Claudio
  • 10,614
  • 4
  • 31
  • 71
xl600
  • 71
  • 5
  • What graphics driver are you using (assuming this has a graphics chipset)? Check with lsmod. Oftentimes, graphics drivers can do operations (I think they're DMA-related) that lock things at a hardware level for multiple milliseconds. I've seen documentation that this is true for Intel's drivers and observed it for Nvidia's proprietary drivers and nouveau. In my experience, the VESA drivers don't have that problem. – crosstalk Aug 12 '18 at 04:22
  • Do you have some simple io’s such as a gpio or a serial port with modem controls; basically something you can hook a scope up to? Instrument the kernel, so that the ipi handler asserts some signal at its start, and de-asserts it at its end. If you have a few such signals available, there are other points it is worthwhile to place similar brackets. Once you find the path that is causing the problem ($5 its ipi), subdivide the path with the same. Eventually you will reduce it to a spin lock, likely on an alloc/free. – mevets Aug 12 '18 at 23:48
  • I’ll check the graphic driver and disable it. I got lazy and installed the os with default options. I’m alread using the parallel port to watch the thread activation and sleep. Lots of other pci and isa based gpio to choose from too (pc/104+). sounds like I have something to do today! Thanks for the suggestions. – xl600 Aug 13 '18 at 12:20
  • 1
    Disabled i915 and things improved only slightly. It wasn’t until I added noacpi to the kernel parameters did things start to settle down. I’m not exactly sure what that will effect, but after some tuning I found I can get a rock steady 3333hz even with disk io and network io under a fair load. Thanks for the ipi clue! That led me to https://help.ubuntu.com/community/ReschedulingInterrupts – xl600 Aug 15 '18 at 00:37
  • Without noacpi, check whether your CPU's frequency is changing. Disabling SpeedStep may not have disabled ACPI P-states (I don't recall whether they're the same thing at the moment). P state transitions can take time – crosstalk Aug 16 '18 at 15:06

1 Answers1

0

SCHED_DEADLINE tasks have higher priority over all the other user-level tasks with different priorities (i.e. SCHED_RR, SCHED_FIFO, SCHED_OTHER).

Note that since kernel 4.16, overruning can be checked by using the SCHED_FLAG_DL_OVERRUN and a user-level handler on the SIGXCPU signal (see here).

If you need to check what the CPU is doing, ftrace is probably the best approach.

Claudio
  • 10,614
  • 4
  • 31
  • 71