how to read multiple serial ports in realtime in C or Python

Question

I need to read multiple (at least 2) serial ports (currently two ports on a FT2232H module connected through USB).

I am using it to monitor a serial connections, so the two ports have their RX connected in parallel to RX and TX of the serial I need to monitor.

Setup is very similar to this.

I am setting up ports like this:

#define waitTime   0

int start_dev(const int speed, const char *dev) {
    int fd = open(dev, O_RDWR | O_NOCTTY |O_NONBLOCK| O_NDELAY);
    int isBlockingMode, parity = 0;
    struct termios tty;

    isBlockingMode = 0;
    if (waitTime < 0 || waitTime > 255)
        isBlockingMode = 1;

    memset (&tty, 0, sizeof tty);
    if (tcgetattr (fd, &tty) != 0) {
        /* save current serial port settings */
        printf("__LINE__ = %d, error %s\n", __LINE__, strerror(errno));
        exit(1);
    }

    cfsetospeed (&tty, speed);
    cfsetispeed (&tty, speed);

    tty.c_cflag = (tty.c_cflag & ~CSIZE) | CS8;     // 8-bit chars
    // disable IGNBRK for mismatched speed tests; otherwise receive break
    // as \000 chars
    tty.c_iflag &= ~IGNBRK;         // disable break processing
    tty.c_lflag = 0;                // no signaling chars, no echo,
                                    // no canonical processing
    tty.c_oflag = 0;                // no remapping, no delays
    tty.c_cc[VMIN]  = (1 == isBlockingMode) ? 1 : 0;            // read doesn't block
    tty.c_cc[VTIME] = (1 == isBlockingMode) ? 0 : waitTime;     // in unit of 100 milli-sec for set timeout value

    tty.c_iflag &= ~(IXON | IXOFF | IXANY); // shut off xon/xoff ctrl

    tty.c_cflag |= (CLOCAL | CREAD);        // ignore modem controls,
                                            // enable reading
    tty.c_cflag &= ~(PARENB | PARODD);      // shut off parity
    tty.c_cflag |= parity;
    tty.c_cflag &= ~CSTOPB;
    tty.c_cflag &= ~CRTSCTS;

    if (tcsetattr (fd, TCSANOW, &tty) != 0) {
        printf("__LINE__ = %d, error %s\n", __LINE__, strerror(errno));
        exit(1);
    }
    return fd;
}

... and currently I have this code for reading (I also tried with select()):

...
    for (running=1; running;) {
        for (int*p=devs; p<end; p++) {
            char b[256];
            int n = read(*p, b, sizeof(b));
            if (n > 0) {
                for (int i=0; i<n; i++) {
                    ...
                }
            }
        }
    }
...

This is obviously highly suboptimal because it doesn't suspend waiting for chars.

Problem is I experience some kind of buffering because when two processes exchange data on a tight loop I often see a few requests together and then the corresponding answers (1b6f is request and 19 is the empty answer):

1b6f
19
1b6f
19
1b6f
19
1b6f
191919
1b6f1b6f1b6f
19191919
1b6f1b6f1b6f1b6f
1b6f1b6f1b6f
191919

I also tried using python (pyserial), but I get similar results.

How should I proceed to ensure correct timings are enforced?

Note: I am not very interested in precise timing, but sequence should be preserved (i.e.: I would like to avoid seeing an answer before the request).

Answer 1

The two serial ports will have buffering - the ordering of arrival of individual characters cannot be determined at the application level. That would require writing your own driver or reducing any buffering to 1 character perhaps - at the risk of overrun.

Even then it could only work if you had a real UART and direct control of it and it had no hardware FIFO. With a Virtual UART implemented as a USB CDC/ACM class driver it is not possible in any case because the real-time UART transactions are lost in the master-slave USB transfers which are entirely different to the way a true UART works. Besides that the FT2232H has internal buffering over which you have no control.

In short, you cannot get real-time sequencing of individual characters on two separate ports in your implementation due to multiple factors, most of which cannot be mitigated.

You have to understand that the FT2232 has two real UARTS and USB device interface presenting as two CDC/ACM devices. It has firmware that buffers and exchanges data between the UART and the USB, and USB exchanges are polled for by the host - in its own sweet time, rate and order. The data is transferrred asynchronoulsy in packets rather than individual characters and recovery of the original time of arrival of any individual character is not possible. All you know is the order of arrival of characters on a single port - you cannot determine the order of arrival between ports. And all that is even before the data is buffered by the host OS device driver.

A hardware solution is probably required, using a microcontroller that, working at the UART level will timestamp and log the arrival of each character on each of two ports, then transfer the timestamped log data to your host (perhaps via USB) where you can then reconstruct the order of arrival from the timestamps.

Answer 2

In my opinion what you're trying to do, which is if I understood correctly a kind of port sniffer to identify the transactions exchanged on a serial link is not feasible with USB-to-serial converters and a conventional OS, unless you're running at slow baudrates.

The USB port will always introduce a certain latency (probably tens of milliseconds), and you'll have to put the unpredictability of the OS on top of that.

Since you have two ports you could try to run two separate threads and timestamp each chunk of data received. That might help improve things but I'm not sure it will allow you to clearly follow the sequence.

If you had real (legacy) serial ports, and a not very loaded OS maybe you could do it somehow.

But if what you want is a serial port sniffer on the cheap you can try something like this solution. If you do forwarding on your ports you'll know what is coming from where at all times. Of course, you need to have access to either side of the communication.

If you don't have that luxury, I guess it would be quite easy to get what you want with almost any kind of microcontroller.

EDIT: Another idea could be to use a dual serial port to USB converter. Since both ports are served by the same chip, somehow I think it's likely that you can follow the sequence with one of those. I have access to this one if you post a full working snippet of your code I can test it next week, if you're curious to know.

Answer 3

I am setting up ports like this:
...
This is obviously highly suboptimal because it doesn't suspend waiting for chars.

In spite of this awareness you use and post this code?
I suspect that this "suboptimal" code that polls the system for data while wasting CPU cycles and consumes the process's time slice is part of the problem. You have not posted a complete and minimal example of the problem, and I have only been able to partially replicate the issue.

On a SBC that has two USARTs, I have a program that is generating "request" and "response" data on the serial ports. The generation program is:

#include <errno.h>
#include <fcntl.h> 
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <termios.h>
#include <unistd.h>

int set_interface_attribs(int fd, int speed)
{
    struct termios tty;

    if (tcgetattr(fd, &tty) < 0) {
        printf("Error from tcgetattr: %s\n", strerror(errno));
        return -1;
    }

    cfsetospeed(&tty, (speed_t)speed);
    cfsetispeed(&tty, (speed_t)speed);

    tty.c_cflag |= (CLOCAL | CREAD);    /* ignore modem controls */
    tty.c_cflag &= ~CSIZE;
    tty.c_cflag |= CS8;         /* 8-bit characters */
    tty.c_cflag &= ~PARENB;     /* no parity bit */
    tty.c_cflag &= ~CSTOPB;     /* only need 1 stop bit */
    tty.c_cflag &= ~CRTSCTS;    /* no hardware flowcontrol */

    /* setup for non-canonical mode */
    tty.c_iflag &= ~(IGNBRK | BRKINT | PARMRK | ISTRIP | INLCR | IGNCR | ICRNL | IXON);
    tty.c_lflag &= ~(ECHO | ECHONL | ICANON | ISIG | IEXTEN);
    tty.c_oflag &= ~OPOST;

    /* fetch bytes as they become available */
    tty.c_cc[VMIN] = 1;
    tty.c_cc[VTIME] = 1;

    if (tcsetattr(fd, TCSANOW, &tty) != 0) {
        printf("Error from tcsetattr: %s\n", strerror(errno));
        return -1;
    }
    return 0;
}


int main(void)
{
    char *masterport = "/dev/ttyS0";
    char *slaveport  = "/dev/ttyS2";
    int mfd;
    int sfd;
    int wlen;

    /* open request generator */
    mfd = open(masterport, O_RDWR | O_NOCTTY | O_SYNC);
    if (mfd < 0) {
        printf("Error opening %s: %s\n", masterport, strerror(errno));
        return -1;
    }
    /*baudrate 115200, 8 bits, no parity, 1 stop bit */
    set_interface_attribs(mfd, B115200);

    /* open response generator */
    sfd = open(slaveport, O_RDWR | O_NOCTTY | O_SYNC);
    if (sfd < 0) {
        printf("Error opening %s: %s\n", slaveport, strerror(errno));
        return -1;
    }
    /*baudrate 115200, 8 bits, no parity, 1 stop bit */
    set_interface_attribs(sfd, B115200);

    /* simple output loop */
    do {
        wlen = write(mfd, "ABCD", 4);
        if (wlen != 4) {
            printf("Error from write cmd: %d, %d\n", wlen, errno);
        }
        tcdrain(mfd);    /* delay for output */

        wlen = write(sfd, "xy", 2);
        if (wlen != 2) {
            printf("Error from write resp: %d, %d\n", wlen, errno);
        }
        tcdrain(sfd);    /* delay for output */

    } while (1);
}

Problem is I experience some kind of buffering because when two processes exchange data on a tight loop I often see a few requests together and then the corresponding answers

You do not clarify what you call a "tight loop", but the above program will generate the "response" 30 milliseconds after a "request" (as measured by a two-channel oscilloscope).

BTW the serial terminal interface is highly layered. Even without the overhead of the external bus used by USB, there is at least the termios buffer and the tty flip buffer, as well as a DMA buffer. See Linux serial drivers

Each USART of the SBC is connected to a FTDI USB-to-RS232 converter (which are part of an old quad-port converter). Note that the USB port speed is only USB 1.1. The host PC for the serial capture is 10-year old hardware running an old Ubuntu distro.

---

An attempt to replicate your results produced:

ABCD
x
y
A
BCD
xy
ABCD
xy
ABCD
xy
A
BCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABC
D
xy
ABCD
xy
ABCD
xy
ABC
D
xy
ABCD
xy
ABCD
xy
ABC
D
xy
ABCD
xy
ABCD
xy
ABC
D
xy
ABCD
xy
ABCD
xy
ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD
xyxyxyxyxyxyxyxyxyxyxyxyxy
ABCD
xy
ABCD
xy
AB
CD
xy
ABCD
xy
ABCD
xy
AB
CD
xy
ABCD
xy
ABCD
x
y
A
BCD
xy
ABCD
xy
ABCD
x
y
AB
CD
xy
ABCD
xy
ABCD
x
y

Only once (about 1.5 seconds after the capture program is started) is there a multi-write capture. (There's even a noticeable pause in output before this happens.) Otherwise every read/capture is of a partial or single/complete request/response.

---

Using a capture program that uses blocking I/O, the results are consistenly "perfect" for a 4-byte request message and 2-byte response message.

ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy

Tweaking the program by changing the VMIN=4 for requests and VMIN=2 for responses to VMIN=1 for everything, changes the quality of the captures slightly:

ABCD
xy
ABCD
x
ABCD
y
ABCD
xy
ABC
xy
D
x
ABCD
y
ABCD
xy
ABCD
xy
ABCD
xy
ABCD
xy
ABC
xy
D
x
ABCD
y

Although partial captures occur, there are never any multiple "messages" per read. The output is smooth and consistent, without any pause as with the nonblocking program.

---

---

The capture program that uses blocking reads is:

#include <errno.h>
#include <fcntl.h> 
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <termios.h>
#include <unistd.h>

int set_interface_attribs(int fd, int speed, int rlen)
{
    struct termios tty;

    if (tcgetattr(fd, &tty) < 0) {
        printf("Error from tcgetattr: %s\n", strerror(errno));
        return -1;
    }

    cfsetospeed(&tty, (speed_t)speed);
    cfsetispeed(&tty, (speed_t)speed);

    tty.c_cflag |= (CLOCAL | CREAD);    /* ignore modem controls */
    tty.c_cflag &= ~CSIZE;
    tty.c_cflag |= CS8;         /* 8-bit characters */
    tty.c_cflag &= ~PARENB;     /* no parity bit */
    tty.c_cflag &= ~CSTOPB;     /* only need 1 stop bit */
    tty.c_cflag &= ~CRTSCTS;    /* no hardware flowcontrol */

    /* setup for non-canonical mode */
    tty.c_iflag &= ~(IGNBRK | BRKINT | PARMRK | ISTRIP | INLCR | IGNCR | ICRNL | IXON);
    tty.c_lflag &= ~(ECHO | ECHONL | ICANON | ISIG | IEXTEN);
    tty.c_oflag &= ~OPOST;

    /* fetch bytes as they become available */
    tty.c_cc[VMIN] = rlen;
    tty.c_cc[VTIME] = 1;

    if (tcsetattr(fd, TCSANOW, &tty) != 0) {
        printf("Error from tcsetattr: %s\n", strerror(errno));
        return -1;
    }
    return 0;
}


int main(void)
{
    char *masterport = "/dev/ttyUSB2";
    char *slaveport  = "/dev/ttyUSB3";
    int mfd;
    int sfd;

    /* open request reader */
    mfd = open(masterport, O_RDWR | O_NOCTTY | O_SYNC);
    if (mfd < 0) {
        printf("Error opening %s: %s\n", masterport, strerror(errno));
        return -1;
    }
    /*baudrate 115200, 8 bits, no parity, 1 stop bit */
    set_interface_attribs(mfd, B115200, 4);

    /* open response reader */
    sfd = open(slaveport, O_RDWR | O_NOCTTY | O_SYNC);
    if (sfd < 0) {
        printf("Error opening %s: %s\n", slaveport, strerror(errno));
        return -1;
    }
    /*baudrate 115200, 8 bits, no parity, 1 stop bit */
    set_interface_attribs(sfd, B115200, 2);

    tcflush(mfd, TCIOFLUSH);
    tcflush(sfd, TCIOFLUSH);

    /* simple noncanonical input loop */
    do {
        unsigned char buffer[80];
        int rdlen;

        rdlen = read(mfd, buffer, sizeof(buffer) - 1);
        if (rdlen > 0) {
            buffer[rdlen] = 0;
            printf("%s\n", buffer);
        } else if (rdlen < 0) {
            printf("Error from read: %d: %s\n", rdlen, strerror(errno));
        } else {  /* rdlen == 0 */
            printf("Timeout from read\n");
        }               

        rdlen = read(sfd, buffer, sizeof(buffer) - 1);
        if (rdlen > 0) {
            buffer[rdlen] = 0;
            printf("%s\n", buffer);
        } else if (rdlen < 0) {
            printf("Error from read: %d: %s\n", rdlen, strerror(errno));
        } else {  /* rdlen == 0 */
            printf("Timeout from read\n");
        }               
    } while (1);
}

This is essentially a dual half-duplex capture on each serial terminal for a request-response dialog. An actual full-duplex dialog cannot be accurately captured/displayed.

---

These results using blocking reads would seem to contradict the other answers that USB-serial converters would buffer and packetize the serial data into unrecognizable byte segments.
Only when I use nonblocking reads do I encounter the "buffering" that you report.

Answer 4

You are making bad use of the VMIN and VTIME c_cc cells. If you read carefully the termios(3) manual page, on a basis of VMIN > 0 && VTIME > 0, the driver will not send the data to the application until a timeout of the VTIME duration is detected. In this case the VTIME parameter is an intercharacter timeout (but it blocks until it receives the first char). I think you misinterpret that case. This was introduced in the driver to handle variable length packet input devices, like mice or network, that can deliver several packets in sequence, to ensure that the buffer will be in synch with the starting of a packet (while handling packet loss). But the operation in that mode is to wait indefinitely for the first char, and then wait up to VTIME tenths of a second to see if another char is received, once the VMIN count is reached, in that case, the driver buffers the char and waits for another timeout. This is made for packets with variable length, and a header, you normally set VMIN as the size of the header, then use an intercharacter timeout to handle lost characters after some timeout. This is not what you tell in your question.

To create a scenario in which you read multiple ports and receive individual chars as soon as you get them, you have to use VMIN == 1, VTIME == 0 so you will get each character as soon as it is received. And to receive the first one you get, independently of which port you receive it from, you need to use select(2) system call, which will block you until some input is available on one of several ports, then look at which port it is, and then do a read(2) of that port. If you want fine timestamps, do a clock_gettime(2) as soon as you return from the select(2) system call (you have not yet read(2) the char, but you know then that it is there, later, once you read it, you can associate the timestamp to the right character and port.

As I see in your question, you have already fought with termios(3) and you have an idea of what you want, read select(2) man page and prepare code to handle with that. If you run in trouble, drop me a comment below, so I'll write some code for you. Remember: VMIN is the minimum number of chars you want to receive, never the maximum (the maximum you put it in the parameter to read(2)), and VTIME is only an absolute timeout, when VMIN == 0 (but you can handle timeouts in select(2), better than in the driver)

This kind of mistake is common, I've passed through it also :)

EDIT

I have developed a simple example to monitor several tty lines (not necessarily two) with the approach indicated here. Just to say that it allows a raspberry pi 2B+ to be used as a serial protocol analyzer, by reading character by character and using the best time granularity approach.