Reading serial data in realtime in Python

Question

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.

The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)

Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)

On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)

The problem:

When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.

Here is the code (I omitted most part of the code, but the loop is the same):

ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
  data_raw = ser.readline()
  print(data_raw)

Anyone knows why pySerial takes so much time to read from the serial port till the end of the line? Any help?

I want to have this in real time.

Thank you

I used ser.read(), to read while a \n was not received, adding character by character to the string as they were received, still the same speed. — Vasco Baptista, Nov 11 '13 at 15:07
Don't wait until a `\n` is received - just print out each character as it arrives. Do you get a sudden flood of characters at once or does each individual character arrive on its own? — Tim, Nov 11 '13 at 15:22
since i was doing ser.read() and then print() i was getting one character per line — Vasco Baptista, Nov 11 '13 at 15:27
@VascoBaptista With the accepted solution (to remove timeout and use `inWaiting()`), it could still be possible that the Python had to wait for all bytes to be received before printing out the entire message (multi-bytes). This would logically cause a delay. Was your system still **real-time** Can you confirm it ? — Pe Dro, Dec 24 '20 at 04:28

Kobi K · Accepted Answer · 2013-11-11T15:07:32.237

42

You can use inWaiting() to get the amount of bytes available at the input queue.

Then you can use read() to read the bytes, something like that:

While True:
    bytesToRead = ser.inWaiting()
    ser.read(bytesToRead)

Why not to use readline() at this case from Docs:

Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.

You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)

edited Nov 11 '13 at 15:07

answered Nov 11 '13 at 14:56

Kobi K

7,743
6
42
86

2

It worked, now I will try to separate all strings and run my code for each one of them. I will give feedback soon – Vasco Baptista Nov 11 '13 at 15:18
Yes I understand, but since my messages are something like: 213531\n 616516\n 516861\n I would think it would read quite fast... anyway my timeout was 2 seconds and the messages were arriving on my script faster than that (printed on screen after readline). Meaning that it was never reaching the timeout. Even adding \r\n did not solve anything. Setting timeout=0 did not change anything, same speed, same everything (except that it was locking when I was turning OFF the PIC) – Vasco Baptista Nov 11 '13 at 15:22
@Vasco Baptista one more thing, `print()` is influenced by the terminal you work with... I'll give you a reference to this [post](http://stackoverflow.com/a/3860319/1982962) try to read it it can help. – Kobi K Nov 11 '13 at 15:40
Thanks for the tip @Kobi K, I know about that, but for the purpose of checking in enough because I am not not printing the messages in my final script. Also the script is now working, i added some extra code to separate the messages i receive and put them in a queue to be processed in other thread. Thanks all for the help – Vasco Baptista Nov 11 '13 at 16:59
So with serial data does it queue up until it is read off the buffer? For instance if you did serial.write(b'100\n') serial.write('b'200\n') serial.write(b'300\n'). Then a serial.readline() would you get >100 and if you continued to do readline it would get the next one in line? – bretcj7 Oct 21 '16 at 18:52
@bretcj7 Yes, `readline()` in ended by CR meaning read a '\n' terminated line – Kobi K Oct 21 '16 at 22:26

score 7 · Answer 2 · answered Jun 17 '19 at 13:55

A very good solution to this can be found here:

Here's a class that serves as a wrapper to a pyserial object. It allows you to read lines without 100% CPU. It does not contain any timeout logic. If a timeout occurs, self.s.read(i) returns an empty string and you might want to throw an exception to indicate the timeout.

It is also supposed to be fast according to the author:

The code below gives me 790 kB/sec while replacing the code with pyserial's readline method gives me just 170kB/sec.

class ReadLine:
    def __init__(self, s):
        self.buf = bytearray()
        self.s = s

    def readline(self):
        i = self.buf.find(b"\n")
        if i >= 0:
            r = self.buf[:i+1]
            self.buf = self.buf[i+1:]
            return r
        while True:
            i = max(1, min(2048, self.s.in_waiting))
            data = self.s.read(i)
            i = data.find(b"\n")
            if i >= 0:
                r = self.buf + data[:i+1]
                self.buf[0:] = data[i+1:]
                return r
            else:
                self.buf.extend(data)

ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)

while True:

    print(rl.readline())

score 5 · Answer 3 · answered Nov 21 '14 at 10:41

You need to set the timeout to "None" when you open the serial port:

ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False)

This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end: line = ser.readline()

Once you have the data, it will return ASAP.

score 3 · Answer 4 · answered Nov 11 '13 at 14:57

3

From the manual:

Possible values for the parameter timeout: … x set timeout to x seconds

and

readlines(sizehint=None, eol='\n') Read a list of lines, until timeout. sizehint is ignored and only present for API compatibility with built-in File objects.

Note that this function only returns on a timeout.

So your readlines will return at most every 2 seconds. Use read() as Tim suggested.

answered Nov 11 '13 at 14:57

msw

42,753
9
87
112

I had problems using readlines. So I tried Kobi K solution. It works, now i just have to parse the received data and separate the strings – Vasco Baptista Nov 11 '13 at 15:17

Reading serial data in realtime in Python

4 Answers4

Linked