Lowest latency notification method between process under Linux

Question

I'm looking for the lowest latency IPC that allow to put one process to sleep and allow other process to wake it.

I'm looking for the lowest latency method. Some possible methods so far:

Writing a byte to a pipe and reading it from it.
Writing a byte to a socket and reading it from it.
Sending a signal (kill) and waiting for it (sigwait)
Using sem_post/sem_wait

Any other better ideas?

Any solution that is Linux specific is fine as well.

Not a complete duplicate, but you may want to look at http://stackoverflow.com/questions/2635272/fastest-low-latency-method-for-inter-process-communication-between-java-and-c which has some benchmarks and pointers to benchmarking tools. — Chris J, Nov 02 '11 at 11:10
https://stackoverflow.com/a/54164058/4123703 would help as well. — Louis Go, Nov 16 '20 at 05:36

score 28 · Accepted Answer · answered Nov 05 '11 at 14:28

Generally... There is almost no difference between the OS methods.

Setup:

Two processes with affinity to two different CPUs.
One process sleeps (nanosleep) for N microseconds measures current time and then notifies other process.
Other process wakes measures current time and compares it to the client's time.
Average, standard deviation, median and percentile 95 is calculated over 1K samples after warm-up on 100 notifications.
OS: Linux 2.6.35 x86_64
CPU: Intel i5 M460

Results:

Semaphore (sem_wait/sem_post - kernel - futex):

sleep us     mean             median      %95
       1     4.98 ±18.7         3.78     5.04
      10     4.14 ±14.8         3.54     4.00
     100    20.60 ±29.4        22.96    26.96
    1000    49.42 ±37.6        30.62    78.75
   10000    63.20 ±22.0        68.38    84.38

Signal (kill/sigwait)

sleep us     mean             median      %95
       1     4.69 ±3.8          4.21     5.39
      10     5.91 ±14.8         4.19     7.45
     100    23.90 ±17.7        23.41    35.90
    1000    47.38 ±28.0        35.27    81.16
   10000    60.80 ±19.9        68.50    82.36

Pipe (pipe + write/read)

sleep us     mean             median      %95
       1     3.75 ±5.9          3.46     4.45
      10     4.42 ±3.5          3.84     5.18
     100    23.32 ±25.6        24.17    38.05
    1000    51.17 ±35.3        46.34    74.75
   10000    64.69 ±31.0        67.95    86.80

Socket (socketpair +write/read)

sleep us     mean             median      %95
       1     6.07 ±3.2          5.55     6.78
      10     7.00 ±7.1          5.51     8.50
     100    27.57 ±14.1        28.39    50.86
    1000    56.75 ±25.7        50.82    88.74
   10000    73.89 ±16.8        77.54    88.46

As a reference busy waiting:

sleep us     mean             median      %95
       1     0.17 ±0.5          0.13     0.23
      10     0.15 ±0.3          0.13     0.19
     100     0.17 ±0.3          0.16     0.21
    1000     0.22 ±0.1          0.18     0.35
   10000     0.38 ±0.3          0.30     0.78

thanks. [My results are the same](https://gist.github.com/1342088#file_results.txt) on `python -mplatform -> Linux-3.0.0-12-generic-x86_64-with-Ubuntu-11.10-oneiric` — jfs, Nov 05 '11 at 22:12
Any explanation as to why sleep duration has so much influence on latency? — Marian, Aug 10 '23 at 07:58

foool · Answer 2 · 2019-11-29T02:13:46.277

Using the same code provided by @Artyom, but in a more morden hardware.

CPU: i9-9900k, closing C/S/P-state and set scaling policy as performance which keeps cores running at the maximum frequency(~5GHz).

OS: Preempt-RT patched Linux with kernel5.0.21, providing better real-time performance.

CPU affinity: two processes apartly running in two isolated cores, which keep away from irrelevant processes and interrupts AMSP.