How to increase speed and split data with multiple plots using PyQtGraph?

Question

I am reading datas from serial port using an STM32 kit. Problem is that I need to use own timestamp for plot ADC datas. That is mean x-axis should be my RTC time(using ms for this) and y-axis is ADC datas. There are programs for plot serial port but as I said I need to set own time for graph. I tried matplotlib for this but it was really slow. Then have used pyqtgraph and this script:

from pyqtgraph.Qt import QtGui, QtCore
import numpy as np
import pyqtgraph as pg
from pyqtgraph.ptime import time
import serial

app = QtGui.QApplication([])

p = pg.plot()
p.setWindowTitle('live plot from serial')
curve = p.plot()

data = [0]
raw=serial.Serial("/dev/ttyACM0",115200)
#raw.open()

def update():
    global curve, data
    line = raw.readline()
    data.append(int(line))
    xdata = np.array(data, dtype='float64')
    curve.setData(xdata)
    app.processEvents()

timer = QtCore.QTimer()
timer.timeout.connect(update)
timer.start(0)

if __name__ == '__main__':
    import sys
    if (sys.flags.interactive != 1) or not hasattr(QtCore, 'PYQT_VERSION'):
        QtGui.QApplication.instance().exec_()

This is slow too but fast compare with mathplotlib. Now I can't find how split my timestamp and ADC datas for plot like x,y. My datas are spliting with ';'.

Thanks for answers.

Edited:

I changed my code reading speed looking enough for know. But know it is plotting some glitches like timetamp is jumping with forward and come back or very big numbers of x-axis datas. I am monitoring datas on a serial port GUI and I can't find any wrong data. Somethings is coming from Python code, i think. Can I ignore these glitches on plotting program?

Code now:

import numpy as np
import pyqtgraph as pg
import serial

app = pg.Qt.QtGui.QApplication([])
p = pg.plot()
p.setWindowTitle('live plot from serial')
curve = p.plot()

data = [0]
tdata = [0]
temp = [0]
datax = [0]
datay = [0]

temp = 0
now = 0
k = 0
raw=serial.Serial("/dev/ttyACM0",115200, timeout=None)
while p.isVisible():
    line = raw.readline().decode('utf-8').strip()
    print("raw line:", line)
    line = str(line)
    print("str line:", line)
    line = line.split(':')
    print("splitted line:", line)
    if len(line) >= 4:
        print("line>4:", line)
        tdata = line[0]
        data = line[1]
        print("line[0]; line[1]:", tdata, line)
        tdata = int(tdata)
        data = int(data)
        print("int(tdata)", tdata)
        print("int(line)", data)
        datax.append(int(tdata))
        datay.append(int(data))
        xdata = np.array(datax, dtype='float64')
        ydata = np.array(datay, dtype='float64')
        p.setXRange(tdata-500, tdata+500, padding=0)
        curve.setData(xdata, ydata)
        # p.setYRange(0 ,data+30, padding=0)
        print("now will refresh the plot")
        app.processEvents()
    else:
        print("line<4:", line)

Could you show an example line of your data so we can discuss how to split it? — Joe, Dec 24 '19 at 18:47
'PRINTF("%d:%d:%d:%d\n",data_s.packetCount, databuffer,dat.frame.sec,dat.frame.subsec);' this is my packet — fury, Dec 24 '19 at 18:53
Thank you again Joe. I want to ask something again. How limits to graph for example only monitor 1000 ms on the screen. It is starting from 0 and keep going in the same screen. For too much sample it is hard to recognize datas. — fury, Dec 25 '19 at 13:33
I my case I was relying on a microcontroller which had a fixed sampling rate. In your case this is probably the STM32, I was using a SAMD21. Then you can calculate the desired number of samples to display and slice them from your numpy array (which is probably larger, e.g. twice the size). — Joe, Dec 25 '19 at 15:47
I am trying too apply a realtime range for like (xdata-1000 , xdata+1000) — fury, Dec 25 '19 at 15:49
If your microcontroller is logging the time or you don't have a fixed sampling rate you can use Numpy's boolean indexing to slice the range you want. see https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#boolean-array-indexing or here https://docs.scipy.org/doc/numpy-1.13.0/user/basics.indexing.html#boolean-or-mask-index-arrays — Joe, Dec 25 '19 at 15:51
For now something like that; `p.setXRange(tdata - 500 ,tdata+500, padding=0)` looking good. But problem is now glitches. Sometimes serial port send very big numbers I should ignore them someway — fury, Dec 25 '19 at 15:53
Depends on your data. If your outliers depend on some sort of integer buffer overflow or all have the same value then you can use some min or max threshold or remove that specific value. Or you could smooth your data using [running mean](https://stackoverflow.com/questions/13728392/moving-average-or-running-mean) or one of the filters listed at https://docs.scipy.org/doc/scipy/reference/signal.html — Joe, Dec 26 '19 at 17:42

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

Splitting the data

When you use Serial.readline() the data read will contain the newline character \n at the end (see Python univeral newlines, could be \r\n\ if you send from Windows).

First decode the bytes received and remove the newline character:

data_string = r.readline().decode('utf-8').strip()

Then split the string at :

data_split = data_string.split(':')

Now data_split is a list with the entries

[packetCount, databuffer, sec, subsec]

and you can convert them to floats or integers and put them in the Numpy array.

Speed improvement

Serial.readline might slow down your code. Use something like this https://stackoverflow.com/a/56632812/7919597 .

Also consider shifting the data in a fixed numpy array instead of creating a new one each time with xdata = np.array(data, dtype='float64').

See Shift elements in a numpy array

I am using these functions in combination with a Thread like

import threading
import queue

q = queue.Queue()
thread = threading.Thread(target=read_from_port, args=(serial_port, q))
thread.start()

to read from the serial port.

There is a very helpful example in the PyQtGraph examples:

https://github.com/pyqtgraph/pyqtgraph/blob/develop/examples/scrollingPlots.py

Is shows three different methods for plotting scrolling plots with Numpy arrays.

I ended up using a combination of method 1 and 3, shifting as many places as were read from the serial port in the meantime.

Thank you very much post seems very helpful. I am trying to learn Python as a newbie so i can't understand very good your thread example but I will check it too. — fury, Dec 24 '19 at 18:14

How to increase speed and split data with multiple plots using PyQtGraph?

1 Answers1

Linked