1

I have a hardware IO task (write/read serial message) that has a strict jitter requirement of less than 200 micro seconds. I need to be able to isolate both a CPU core/s and hardware/interrupt.

I have tried 2 things that have helped but not gotten me all the way there.

  1. Using <termios.h> to configure the tty device. Specifically setting VMIN=packet_size and VTIME=0
  2. Isolcpus kernel argument in /etc/default/grub and running with taskset

I am still seeing upwards of 5 ms (5000 us) of jitter on my serial reads. I tested this same application with pseudo serial devices (created by socat) to eliminate the HW variable but am still seeing high jitter.

My test application right now just opens a serial connection, configures it, then does a while loop of writes/reads.

Could use advice on how to bring jitter down to 200 us or less. I am considering moving to a dual boot RTOS/Linux with shared memory, but would rather solve on one OS.

Real Application description:

  1. Receive message from USB serial
  2. Get PTP (precision time protocol) time within 200 us of receiving the first bit
  3. Write packet received along with timestamp to shared memory buffer shared with a python application: <timestamp, packet>.
  4. Loop.

Also on another isolated HW/core:

  1. Read some <timestamp, packet> from a shared memory buffer
  2. Poll PTP time until <timestamp>
  3. Transmit <packet> at within 200 us of <timestamp> over USB serial
  4. Loop
Lenna
  • 1,220
  • 6
  • 22
  • Do you really want to do it over USB? It adds a lot of jitter on its own. Can you use a direct serial line, or RS422/RS485? Also, don't sleep, better poll and spin. – SK-logic Oct 07 '21 at 09:43
  • 1
    What freedom do you have on the hardware side? Linux UART is pitiful. Especially on x86. For your latency requirements a more reasonable hardware would have been a better choice. E.g., an FPGA SoC such as CYCLONE V or Zynq UltraScale+, with UARTs implemented on PL, communicating lock-less and without interrupts with a user space program on Linux PS. Another option is to use an intermediate MCU talking UART and then continuously communicating with the Linux host over SPI (again, not very convenient on x86). In general, x86 platforms suck and are not suitable for RT comms. – SK-logic Oct 07 '21 at 11:04
  • 1
    The issue here is that there are python processes running on a linux OS that have to transmit/receive realtime serial messages. I need to transfer timely data between RT serial and python process. – Lenna Oct 07 '21 at 11:21
  • For the purposes of the constraints of the question, lets ignore real HW (/dev/tty*) and instead solve the pseudo tty jitter problem. It simplifies it a bit – Lenna Oct 07 '21 at 11:23
  • 1
    I would desperately like to make this work on linux. I have PREEMPT_RT compiled in, and have found some discussions online about “tickless mode” but have gotten no noticeable speedup from preempt RT – Lenna Oct 07 '21 at 12:03
  • With SoCs I mentioned you can easily do it, they still run Linux, but you access the UART data directly without a single system call and without context switches. See Pynq for example of how it's done directly from Python. I doubt you can get any kind of a real-time guarantee from /dev/tty devices, with all the unavoidable kernel calls and context switches. – SK-logic Oct 07 '21 at 12:53
  • 1
    For the external MCU talking over SPI option - e.g., if your Linux box is a Rasbperry Pi, it's got a DMA SPI mode, so, again, you can get your serial data without ever needing to context-switch. – SK-logic Oct 07 '21 at 12:57
  • Hrmm I have a few raspis laying around, I’ll see what I can do there and post a full solution if/when I figure this out – Lenna Oct 08 '21 at 00:15
  • 1
    Ok, should be easy then. You'll also need any MCU board, nucleo, for.example, to translate UART transactions into SPI stream, which RPi must poll continuously (dedicate one CPU core.fully to it). Then you'll get around Linux deficiencies. – SK-logic Oct 08 '21 at 06:52

1 Answers1

1

To reduce process latency/jitter:

  1. Isolate some cores /etc/default/grub... isolcpus=0

  2. Never interrupt RT tasks set /proc/sys/kernel/sched_rt_runtime_us to -1

  3. Run high priority on isolated core schedtool -a 0 -F -p 99 -n -20 -e $CMD

To reduce serial latency/jitter

  1. File descriptor options O_SYNC
  2. Ioctl ASYNC_LOW_LATENCY
  3. Termios VMIN = message size and VTIME = 0
  4. Use tcdrain after issuing write commands
Lenna
  • 1,220
  • 6
  • 22