Parse int values with pyserial and plot a histogram

Question

I have been working on this code for hours, and because of my lack of python knowledge, I am not getting very far. I have large amount of data coming in from an Arduino. (sample serial data below)

'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00171,170,171,171,171,172,171,171,172,171,170,168,169,170,170,170,168,170,171,170,170,170,170,170,171,170,170,170,169,169,169,170,169,168,168,167,168,168,169,167,168,168,168,168,168,167,167,168,169,168,169,169,168,169,167,167,167,167,167,166,167,167,167,167,166,167,167,166,166,167,167,167,167,167,166,166,167,166,166,166,166,166,166,165,166,166,166,165,166,165,166,166,165,166,166,166,165,165,166,165,165,165,165,165,166,166,165,165,165,165,166,165,165,165,165,165,165,165,165,166,166,165,165,165,162,167,165,166,166,165,164,167,166,165,165,165,164,166,165,164,164,164,165,165,164,165,164,164,164,164,164,164,165,164,164,164,164,164,164,163,164,164,164,163,164,164,164,164,164,164,164,163,164,164,164,164,163,164,164,164,163,164,164,163,163,163,163,163,163,164,163,163,163,163,163,163,163,163,163,164,163,163,163,163,163,163,162,163,163,163,163,162,163,163,163,163,163,163,163,163,163,163,163,163,163,162,163,162,162,163,162,163,162,162,162,162,163,163,162,162,162,162,162,163,162,163,163,162,163,162,162,162,162,162,163,162,162,160,162,162,163,162,162,162,165,163,162,162,162,162,162,162,162,162,162,161,162,161,162,162,162,161,161,162,161,161,161,161,\n'

As you can see, there is "junk" in the beginning. Here is my code thus far:

import serial
import matplotlib.pyplot as plt
import numpy as np
import time

serialArduino = serial.Serial('/dev/ttyUSB0', 9600, timeout=3)

plt.ion()
cnt=0

ydata = [0] * 290
line, = plt.plot(ydata)
plt.ylim([0,1000])

while True:
  bytesToRead = serialArduino.inWaiting()
  valueRead = serialArduino.readline().rstrip()
  try:
     list = int(valueRead)
  except ValueError as e:
     list = 0 # ????
 #   mylist = list.partition("\n")[0]
  if list <=1000:
    ydata.append(list)
    del ydata[0]
    line.set_xdata(np.arange(len(ydata)))
    line.set_ydata(ydata)
    plt.axis([ 0, 288 , 0, 1000])
    binBoundaries = np.linspace(0, 288, 1.77)
    plt.show()
    time.sleep(1)

My biggest issues are getting rid of the "junk" and splitting the "good numbers" from the commas so that they can be interpreted as separate numbers (do I even need to do this?). I think the \n is taken care of. I appreciate any help or advice or links you can give me.

EDIT: Some indentations messed up when I pasted. Also altered some code.

Code for Arduino (Pro Trinket 5V/ 16MHz (FTDI))

    /*
 * Macro Definitions
 */
#define SPEC_TRG         A0
#define SPEC_ST          A1
#define SPEC_CLK         A2
#define SPEC_VIDEO       A3

#define SPEC_CHANNELS    288 // New Spec Channel
uint16_t data[SPEC_CHANNELS];

void setup(){

  //Set desired pins to OUTPUT
  pinMode(SPEC_CLK, OUTPUT);
  pinMode(SPEC_ST, OUTPUT);

  digitalWrite(SPEC_CLK, HIGH); // Set SPEC_CLK High
  digitalWrite(SPEC_ST, LOW); // Set SPEC_ST Low

  Serial.begin(9600); // Baud Rate set to 9600

}

/*
 * This functions reads spectrometer data from SPEC_VIDEO
 * Look at the Timing Chart in the Datasheet for more info
 */
void readSpectrometer(){

  int delayTime = 1; // delay time

  // Start clock cycle and set start pulse to signal start
  digitalWrite(SPEC_CLK, LOW);
  delayMicroseconds(delayTime);
  digitalWrite(SPEC_CLK, HIGH);
  delayMicroseconds(delayTime);
  digitalWrite(SPEC_CLK, LOW);
  digitalWrite(SPEC_ST, HIGH);
  delayMicroseconds(delayTime);

  //Sample for a period of time
  for(int i = 0; i < 3000; i++){

      digitalWrite(SPEC_CLK, HIGH);
      delayMicroseconds(delayTime);
      digitalWrite(SPEC_CLK, LOW);
      delayMicroseconds(delayTime); 

  }

  //Set SPEC_ST to low
  digitalWrite(SPEC_ST, LOW);

  //Sample for a period of time
  for(int i = 0; i < 85; i++){

      digitalWrite(SPEC_CLK, HIGH);
      delayMicroseconds(delayTime);
      digitalWrite(SPEC_CLK, LOW);
      delayMicroseconds(delayTime); 

  }

  //One more clock pulse before the actual read
  digitalWrite(SPEC_CLK, HIGH);
  delayMicroseconds(delayTime);
  digitalWrite(SPEC_CLK, LOW);
  delayMicroseconds(delayTime);

  //Read from SPEC_VIDEO
  for(int i = 0; i < SPEC_CHANNELS; i++){

      data[i] = analogRead(SPEC_VIDEO);

      digitalWrite(SPEC_CLK, HIGH);
      delayMicroseconds(delayTime);
      digitalWrite(SPEC_CLK, LOW);
      delayMicroseconds(delayTime);

  }

  //Set SPEC_ST to high
  //digitalWrite(SPEC_ST, HIGH);

  //Sample for a small amount of time
  for(int i = 0; i < 7; i++){

      digitalWrite(SPEC_CLK, HIGH);
      delayMicroseconds(delayTime);
      digitalWrite(SPEC_CLK, LOW);
      delayMicroseconds(delayTime);

  }

  digitalWrite(SPEC_CLK, HIGH);
  delayMicroseconds(delayTime);

}

/*
 * The function below prints out data to the terminal or 
 * processing plot
 */
void printData(){

  for (int i = 0; i < SPEC_CHANNELS; i++){
    Serial.print(data[i]);
    Serial.print(',');

  }

  Serial.print("\n");
}

void loop(){

  readSpectrometer();
  printData();
  delay(1000);  

}

The code looks fine, I'll try to test it next week to see whether I can reproduce the garbage output. :) — Patrick Trentin, Feb 14 '17 at 14:10

score 0 · Accepted Answer · edited May 23 '17 at 12:01

Note: I focus only on the problem of obtaining a list of data points from Arduino. I think that how to plot this data should be an entirely different question, especially if you want to do it live. (in this regard there are already several questions covering this problem here on StackOverflow, e.g. see this and this)

This is what I came up on the spot, without any testing:

#!/usr/bin/env python2.7

import serial
import matplotlib.pyplot as plt
import numpy as np
import time

raw_buffer = ''
ydata = []
ydata_changed = False
ser = serial.Serial('/dev/ttyUSB0', 9600, timeout=None)
                                          # no timeout!

while True:

    # read only available data
    waiting = ser.inWaiting()
    if waiting > 0:

        # append new data to incoming buffer, remove 'junk'
        raw_buffer = raw_buffer + \
                     ser.read(waiting).replace('\x00', '').replace('\n', '')

        # Take care of 'partial' numbers
        # (e.g. raw_buffer='...,171,16' where '16'
        # should be '162' but the extra '2' is
        # still waiting in the input buffer
        idx = raw_buffer.rfind(',')
        raw_data = raw_buffer[0:idx]
        raw_buffer = raw_buffer[idx + 1:]

        # get list of new values and store them safely
        raw_list = map(lambda x: int(x), \
                   filter(lambda x: x != '', raw_data.split(',')))

        if len(raw_list) > 0:
            ydata.extend(raw_list)
            ydata_changed = True

    if ydata_changed:
        ydata_changed = False
        print "New Data: " + str(ydata)

        # do something with ydata

    time.sleep(1)

The code has inline comments and should be otherwise self-explanatory. Note that this code will keep running forever, printing the full content of ydata each time that this has been extended in some way.

To conclude, let me add an observation: plt.show() is a blocking call, so if you insert that instruction within the loop you will not be able to parse new data up until when you close the figure. You might want to either:

set an arbitrary threshold to len(ydata) after which you break from the loop and plot your content
perform serial read and plotting on two different, but communicating, threads

I will give this a go. Thanks so much for you help! I see that you replaced the \x00. Is there a way to get rid of it if the \x00 is more random? (\x44 \x56 \x22... etc) Is there something else I should look into for getting rid of "random junk" that is always at the beginning. (I'm noticing that sometimes it isn't perfectly \x00... ) I am new to python. So I don't know of a code to say "start reading data after ":" (colon). — keakins13, Feb 14 '17 at 12:37
\x82\x9ab\x92\x82\x92b\x92\x82\x9ab\x92\x82\xa2b\x92\x82\x9ab\x92\x82\xa2b\x92\x82\x9ab\x92\x82\x92b\x92\x82\x9ab\x9 — keakins13, Feb 14 '17 at 12:43
@keakins13 as far as I know, *Arduino* shouldn't print any *junk* by itself, though I might be wrong. If you would be so kind to *edit* your question and add the *Arduino* sketch so that I could take a look at it, that would be appreciated. -- Yes, you could `import re` and then use `re.sub('[^0-9a-zA-Z,]+', '', raw_serial)`, where `raw_serial = ser.read(waiting)`. However, this is a bit *brutal*, and since I don't really know what is happening on the *Arduino* side I don't know whether it's going to do more help than damage. — Patrick Trentin, Feb 14 '17 at 12:51

Parse int values with pyserial and plot a histogram

1 Answers1