Issues with Serial Port Program Parsing Data

Question

I am trying to parse data continuously coming in via serial port connection. I am having difficulty printing variable 'extracted_vals' in the correct format.

import serial
import time
import re
ser = serial.Serial( #drop down to change name or baudrate etc. 
    port='name',\
    baudrate=9600,\
    parity=serial.PARITY_NONE,\
    stopbits=serial.STOPBITS_ONE,\
    bytesize=serial.EIGHTBITS,\
        timeout=0)
print("connected to: " + ser.portstr)
extracted_vals = []

while True:
    for line in ser.readline():
        time.sleep(.1)
        signal = chr(line)
        regex = "(?:.*\s)?\nX\n=\n(?P<X>-?\d+)\s\nY\n=\n(?P<Y>.*)"
        parts = signal.split(r'\n\n\n')
        for part in parts:
            m = re.match(regex, part)
            if m is None:
                continue
            X,Y = m.groupdict().values()
            extracted_vals.append(dict(X=int(X),
                                       Y=[Y[i:i+2] for i in range(0,len(Y),2)]))
            print(extracted_vals)
     
ser.close()

When I print(signal),I see a continuous output in the form:

0
0
0
:
0
0

B
K
O
0
0
D
0

X
=
-
9
0

Y
=
A
3
4
2
D
5
S
3
9
8



X
=
-
1
0
1

Y
=

I first want to extract substrings in the form 'X=-98 Y=FF32A4DE'. The substrings repeat in the continuous output above and each is unique. As the serial port in connected the data just keep coming in continuously, I am thinking I maybe need some sort of time interval loop to extract substrings stepwise and parse the data.

I want the output of print(extracted_vals) to look like, with X as integers:

[{'X': -90, 'Y': ['43', '01', 'BB', 'C6', '2D', '87', 'E4', '68']}, {'X': -74, 'Y': ['43', '21', '67']}, {'X': -88, 'Y': ['CF', 'BA','03']}]

I'd suggest that you first make a simple code (no regex, no fancy stuff) that just prints what you receive, so you verify that you are receiving what you expect, without "irrelevant characters" or other strange things.Once you are confident 100% that you get what you expect your problem is no longer about receiving data via serial port, but rather postprocessing data into a dictionary. One thing at a time. — , Mar 21 '22 at 11:10
Thanks for the feedback, appreciate the advice. I get what you are saying, I am receiving what I expect with the code stripped down and printing the raw data. My main difficulty at the moment is extracting substrings in the form 'X=-80 Y=F2A412AA', and getting them outputted in the format I need. — Huramaki, Mar 21 '22 at 11:17
ok, then could you update your question? it looks like you just have an input string that you want to split to extract keys and values. Using space and `=` as delimiters should do the job. — , Mar 21 '22 at 11:19
I updated my question, perhaps I laid it out in an unclear way. The string that looks like an input is the raw data that gets continuously outputted, if printed. I need to try and extract the parts with X and Y in the form of my previous comment before parsing, and the final block of text is what I want the output to look like. — Huramaki, Mar 21 '22 at 11:32

kunif · Answer 1 · 2022-03-21T13:40:30.763

Since the loop processing for line in ser.readline(): is performed, it seems that the line contains only one character instead of one line, and the desired processing is not performed.
And maybe you don't need a variable called signal.

Why not process the read instead of a for loop as follows?
Please try to see if that changes anything.

while True:
    line = ser.readline()          #### change from for loop to method call
    time.sleep(.1)
    regex = "(?:.*\s)?\nX\n=\n(?P<X>-?\d+)\s\nY\n=\n(?P<Y>.*)"
    parts = line.split(r'\n\n\n')  #### change from signal to line
    for part in parts:
        m = re.match(regex, part)
        if m is None:
            continue
        X,Y = m.groupdict().values()
        extracted_vals.append(dict(X=int(X),
                                   Y=[Y[i:i+2] for i in range(0,len(Y),2)]))
        print(extracted_vals)

For example, these articles will be helpful.

An article about disassembling what you read into character units and processing it.
Python Serial: How to use the read or readline function to read more than 1 character at a time

An article that processes what is read line by line.
readline() in pySerial sometimes captures incomplete values being streamed from Arduino serial port

For comment:

I'm sorry, I didn't read the source code in detail.
As @Sembei Norimaki commented, why not print out the following read data without processing it and check it?

while True:
    line = ser.readline()
    time.sleep(.1)
    print(line)

Or if it's some standardized data format, why not get the document and look it up, or look for a library that does the work?

Thank you for your answer, and for your advice. I tried the line = ser.readline() and its output is many repetitions of """b'' b'' b''""" I will look into those links you posted I am sure they will help. — Huramaki, Mar 21 '22 at 13:08

Issues with Serial Port Program Parsing Data

1 Answers1