how to make matplotlib update from continuous data source?

Question

I am trying to make a simple graph from a data gathered from a continuous real-time data source.

My code for using matplotlib is below:

import matplotlib.pyplot as plt
import matplotlib.animation as animation
import time
from serialdata import SerialData 

fig = plt.figure()
ax1 = fig.add_subplot(1,1,1)

def animate(i):
    xar = []
    yar = []

    #Open Serial Port and Receive Continuous Data 
    #in format of number,number
    a = SerialData()
    b = a.setSerial('COM3', 9600)
    dataArray = a.getSerial(9999)

    for eachLine in dataArray:
        if len(eachLine)>1:
            x,y = eachLine.split(',')
            xar.append(int(x))
            yar.append(int(y))
    ax1.clear()
    ax1.plot(xar,yar)
ani = animation.FuncAnimation(fig, animate, interval=1000)
plt.show()

The serialdata.py just yields the data every time it gets from the data source:

import serial
from time import sleep

class SerialData:
    def __init__(self):
        pass        

    def setSerial(self, port, baudrate):
        self.port = port
        self.baudrate = baudrate
        print("Opening Serial Port...")
        self.ser = serial.Serial(self.port, self.baudrate, timeout=1)
        sleep(2)
        print("Setup Successful") 

    def getSerial(self, read):
        while True:
            self.data = self.ser.read(read)
            if len(self.data)>0:
                yield self.data.decode('utf-8')
            sleep(.1)
        self.ser.close() 

supposedly it should send data in form of:

1,2
2,5
3,7
4(autoincrement),5(random number)

and works just fine when I just make them print on the CLI. However I cannot make it work with the matplotlib.

There is no specific error.

It just shows

Opening Serial Port...
Setup Successful

and... thats it. nothing happens then. What is wrong with my code?

---

I did more research and found that I shouldn't be using show() so I rewrote my code as following:

import time
import numpy as np
import matplotlib.pyplot as plt
from serialdata import SerialData   

plt.axis([0, 1000, 0, 1])
plt.ion()
plt.show()

for i in range(1000):
    # y = np.random.random()

    a = SerialData()
    b = a.setSerial('COM3', 9600)
    dataArray = a.getSerial(9999)

    print("Data Gathering...")
    for eachLine in dataArray:

        if len(eachLine)>1:
            y = eachLine.split(',')[1]

            plt.scatter(i, y)
            plt.draw()
            time.sleep(0.05)

But, the result is the same.

Answer 1

I don't have a serial port, but this is what I tried to do to figure it out:

import matplotlib.pyplot as plt
import matplotlib.animation as animation
import time

import random
from time import sleep

class MockData():
    def __init__(self, n):
        self.n = n  #number of data points to return

    def getData(self):
        start = 0 #mock autoincrement
        for i in range(self.n):
            yield (start, random.randint(0, 100)) #yield a tuple, not a str (x(autoincrem),y(random))
            start+=1  


fig = plt.figure()
ax1 = fig.add_subplot(1,1,1)

def animate(i):
    xar = []
    yar = []

    #Open Serial Port and Receive Continuous Data 
    #in format of number,number
    a = MockData(10)
    dataArray = a.getData() #this is a generator that will yield (x,y) tuples 10 times. 

    for eachLine in dataArray:
            x,y = eachLine
            xar.append(int(x))
            yar.append(int(y))
    ax1.clear()
    ax1.plot(xar,yar)

ani = animation.FuncAnimation(fig, animate, interval=1000)
plt.show()

This code will fail\be very slow:

1. If I request a large amount of data at a time: a=MockData(1000) will execute for ~2-3seconds per frame

2. If readout times are long i.e.

   def getData(self):
       start = 0
       for i in range(self.n):
           yield (start, random.randint(0, 100))
           start+=1
           time.sleep(1)
   

   this will execute ~10 seconds per frame

3. Both

So as far as I can conclude issue is in your SerialData class, specifically in the getSerial method. I know that because that's the method that's responsible for retrieving the actual data. Since I don't have a serial port I can't exactly test where, but I can wager a few guesses.

   def getSerial(self, read):
        while True:
            self.data = self.ser.read(read)
            if len(self.data)>0:
                yield self.data.decode('utf-8')
            sleep(.1)
        self.ser.close()

1. self.data = self.ser.read(read) Issue here is that you request 9999 bytes to be read. 9600 bauds is about 1200 bytes/s. To read 9 999 bytes it would take ~10seconds. And that's if there's actually 9999 new bytes to be read. If there weren't the read function would continue waiting until it read in that amount. This is equal to my 1) test case, except the waiting period would be sleep(10). The if check therefore is already in the read function so you don't have to check again.

2. self.data.decode('utf-8') Check out how long it takes to decode 9999 bytes:

   >>> from timeit import Timer
   >>> ts = Timer("s.decode('utf-8')", "s = b'1'*9999")
   >>> ts.timeit()
   2.524627058740407
   

   Now, granted, this isn't the same conversion as yours, but since I don't have a serial port on my laptop I can't test it. Anyhow, it seems like it's very slow. Right now, you have my case 2) with sleep(12)

3. sleep(.1) this now just seems like adding an insult to injury.

Your code doesn't report an error because it works, it just takes over 3 minutes to read in the first set of data and plot it.

My recommendation is that you ignore this approach, read in data, almost literally, byte per byte, plot them as they come. Have an array in which you just append new bytes as they come and plot that. You could easily pull something like:

serialPort = serial.Serial(self.port, self.baudrate, timeout=1)
data = list() #some global list to append to

def animate():
    d = serialPort.read(2) #this will hang until it completes
    data.append(d) #does the data really have to be in string? 
    ax.plot(data)

ani = animation.FuncAnimation(fig, animate, interval=1000) #optionally send in repeat_delay?
plt.show()

though part about this is that I'm not sure how animation behaves if the data array starts getting really big, so you might consider shifting your x axis to the right, deleting your old data and creating a new array periodically.

Hopefully I was of some help in pointing out which things you need to test for. Start timing all your read-ins from serial port to get a feeling for how long do things take. Don't forget to make sure you're actually reading anything at all from the serial port also, if you're not, the serial.read will just wait there.

I've also never done anything like this, so it's possible I'm way off course. Last time I dealt with serial ports was robotics championship from elementary (rofl).