Low latency serial communication on Linux

Question

I'm implementing a protocol over serial ports on Linux. The protocol is based on a request answer scheme so the throughput is limited by the time it takes to send a packet to a device and get an answer. The devices are mostly arm based and run Linux >= 3.0. I'm having troubles reducing the round trip time below 10ms (115200 baud, 8 data bit, no parity, 7 byte per message).

What IO interfaces will give me the lowest latency: select, poll, epoll or polling by hand with ioctl? Does blocking or non blocking IO impact latency?

I tried setting the low_latency flag with setserial. But it seemed like it had no effect.

Are there any other things I can try to reduce latency? Since I control all devices it would even be possible to patch the kernel, but its preferred not to.

---- Edit ----

The serial controller uses is an 16550A.

What type of serial interface are you using? USB/serial interfaces can be kind of slow. — , Oct 29 '12 at 17:07
you need to check where the 10 ms are spent, because if they are lost by the other device you cannot optimize more than that. — Ottavio Campana, Oct 29 '12 at 17:19
What are sizes of request and answer messages? If both are more than 100 bytes, then you can not get round trip time <10ms (with 115200). — SKi, Oct 29 '12 at 17:26
115200 is extremely slow, so you are guaranteed big latency just moving the bytes across. Your best bet will be bumping the baud to something like 921600. Or even better, switch to gigabit ethernet. — TJD, Oct 29 '12 at 17:43
@OttavioCampana at the moment the time is spend waiting for input. I'm polling until ioctl tells my that input is available and than I read it. — JustMaximumPower, Oct 30 '12 at 06:54
@JustMaximumPower if time is spent it lost, the it's the device which is slow. — Ottavio Campana, Oct 30 '12 at 07:54

score 11 · Answer 1 · edited Mar 21 '22 at 01:53

Request / answer schemes tends to be inefficient, and it shows up quickly on serial port. If you are interested in throughtput, look at windowed protocol, like kermit file sending protocol.

Now if you want to stick with your protocol and reduce latency, select, poll, read will all give you roughly the same latency, because as Andy Ross indicated, the real latency is in the hardware FIFO handling.

If you are lucky, you can tweak the driver behaviour without patching, but you still need to look at the driver code. However, having the ARM handle a 10 kHz interrupt rate will certainly not be good for the overall system performance...

Another options is to pad your packet so that you hit the FIFO threshold every time. It will also confirm that if it is or not a FIFO threshold problem.

10 msec @ 115200 is enough to transmit 100 bytes (assuming 8N1), so what you are seeing is probably because the low_latency flag is not set. Try

setserial /dev/<tty_name> low_latency

It will set the low_latency flag, which is used by the kernel when moving data up in the tty layer:

void tty_flip_buffer_push(struct tty_struct *tty)
{
         unsigned long flags;
         spin_lock_irqsave(&tty->buf.lock, flags);
         if (tty->buf.tail != NULL)
                 tty->buf.tail->commit = tty->buf.tail->used;
         spin_unlock_irqrestore(&tty->buf.lock, flags);
 
         if (tty->low_latency)
                 flush_to_ldisc(&tty->buf.work);
         else
                 schedule_work(&tty->buf.work);
}

The schedule_work call might be responsible for the 10 msec latency you observe.

The reason i use a request/answer scheme is to control access to the bus, since there can be many devices connect to it. Padding the packets had no effect on latency — JustMaximumPower, Oct 30 '12 at 12:19
@JustMaximumPower : try to compile setserial for your platform, and do a setserial /dev/ttySx low_latency — shodanex, Oct 30 '12 at 15:33
reading the documentation of low_latency it seems to be exactly what I need. However it has no impact on my program. I have a theory why I need to test. I will get back to you. — JustMaximumPower, Oct 31 '12 at 08:11
The function `tty_flip_buffer_push` is no longer checking a `low_latency`-flag, and will always call `schedule_work`. I don't know when this behavior got changed. — Ruud Althuizen, Jun 01 '22 at 10:05

JustMaximumPower · Accepted Answer · 2014-04-29T07:38:34.610

9

Having talked to to some more engineers about the topic I came to the conclusion that this problem is not solvable in user space. Since we need to cross the bridge into kernel land, we plan to implement an kernel module which talks our protocol and gives us latencies < 1ms.

--- edit ---

Turns out I was completely wrong. All that was necessary was to increase the kernel tick rate. The default 100 ticks added the 10ms delay. 1000Hz and a negative nice value for the serial process gives me the time behavior I wanted to reach.

edited Apr 29 '14 at 07:38

answered Nov 15 '12 at 10:31

JustMaximumPower

1,257
3
11
22

Any link or document to follow for the same. I am also looking for faster than 10ms serial data communication over USB based virtual serial port. Receiving data from a microcontroller which needs to be responded back ASAP within 100 uS or so. I can try with 1ms. – Rick2047 Nov 20 '19 at 13:14
1

@Rick2047 Here is some documentation on Kernel Tick rate https://elinux.org/Kernel_Timer_Systems . Keep in mind that this question/answer is from 2012. A lot has changed since then. I'm not sure how applicable this still is with tickless kernels. – JustMaximumPower Nov 21 '19 at 05:49

score 7 · Answer 3 · answered Jan 22 '13 at 16:21

Serial ports on linux are "wrapped" into unix-style terminal constructs, which hits you with 1 tick lag, i.e. 10ms. Try if stty -F /dev/ttySx raw low_latency helps, no guarantees though.

On a PC, you can go hardcore and talk to standard serial ports directly, issue setserial /dev/ttySx uart none to unbind linux driver from serial port hw and control the port via inb/outb to port registers. I've tried that, it works great.

The downside is you don't get interrupts when data arrives and you have to poll the register. often.

You should be able to do same on the arm device side, may be much harder on exotic serial port hw.

score 4 · Answer 4 · answered Apr 19 '17 at 13:00

4

Here's what setserial does to set low latency on a file descriptor of a port:

ioctl(fd, TIOCGSERIAL, &serial);
serial.flags |= ASYNC_LOW_LATENCY;
ioctl(fd, TIOCSSERIAL, &serial);

answered Apr 19 '17 at 13:00

Kuba hasn't forgotten Monica

95,931
16
151
313

I'm doing this in code, and it changes the latency timer (as seen with `cat`), but doesn't actually affect the latency. Do I need to close and open the port, or do anything after setting the flag? – Patrick May 02 '17 at 04:07
You shouldn't need to do anything else. – Kuba hasn't forgotten Monica May 02 '17 at 12:11
1

It appears you need to run the code as root for it to work. – Patrick May 03 '17 at 01:14
2

@Patrick That's what one gets for not checking the return codes :) – Kuba hasn't forgotten Monica May 03 '17 at 06:59
2

The `serial_struct` is in ``, BTW. – Alnitak Jul 14 '17 at 21:16

score 2 · Answer 5 · edited Apr 10 '19 at 06:55

In short: Use a USB adapter and ASYNC_LOW_LATENCY.

I've used a FT232RL based USB adapter on Modbus at 115.2 kbs.

I get about 5 transactions (to 4 devices) in about 20 mS total with ASYNC_LOW_LATENCY. This includes two transactions to a slow-poke device (4 mS response time).

Without ASYNC_LOW_LATENCY the total time is about 60 mS.

With FTDI USB adapters ASYNC_LOW_LATENCY sets the inter-character timer on the chip itself to 1 mS (instead of the default 16 mS).

I'm currently using a home-brewed USB adapter and I can set the latency for the adapter itself to whatever value I want. Setting it at 200 µS shaves another mS off that 20 mS.

score 1 · Answer 6 · answered Oct 29 '12 at 17:54

1

None of those system calls have an effect on latency. If you want to read and write one byte as fast as possible from userspace, you really aren't going to do better than a simple read()/write() pair. Try replacing the serial stream with a socket from another userspace process and see if the latencies improve. If they don't, then your problems are CPU speed and hardware limitations.

Are you sure your hardware can do this at all? It's not uncommon to find UARTs with a buffer design that introduces many bytes worth of latency.

answered Oct 29 '12 at 17:54

Andy Ross

11,699
1
34
31

Ok it starts to look like your right. The board uses an 16550A chip which has a receiving FIFO and triggers the interrupt after 14 bytes have been received. The question now becomes how can I change the FIFO size or work around it? – JustMaximumPower Oct 30 '12 at 07:59
Is this just a PC serial port? If so, you might dig around in the linux driver (looks like it's in drivers/tty/serial/8250) to see if there is an ioctl providing programmatic control over the FIFO depths. – Andy Ross Oct 30 '12 at 15:59

score 0 · Answer 7 · answered Oct 30 '12 at 10:15

At those line speeds you should not be seeing latencies that large, regardless of how you check for readiness.

You need to make sure the serial port is in raw mode (so you do "noncanonical reads") and that VMIN and VTIME are set correctly. You want to make sure that VTIME is zero so that an inter-character timer never kicks in. I would probably start with setting VMIN to 1 and tune from there.

The syscall overhead is nothing compared to the time on the wire, so select() vs. poll(), etc. is unlikely to make a difference.

Low latency serial communication on Linux

7 Answers7

Linked

Related