read and stock various data from various usb devices in python

Question

I am a beginner in python, and I am trying to read the data from several sensors (humidity, temperature, pressure sensors...) that I connect with a usb hub to my computer. My main goal is to record every five minutes the different values of those sensors and then store it to analyse it.

I have got all the data sheets and manuals of my sensors (which are from Hygrosens Instruments), I know how they work and what kind of data they are sending. But I do not know how to read them. Below is what I tried, using pyserial.

import serial #import the serial library
from time import sleep #import the sleep command from the time library
import binascii

output_file = open('hygro.txt', 'w') #create a file and allow you to write in it only. The name of this file is hygro.txt

ser = serial.Serial("/dev/tty.usbserial-A400DUTI", 9600) #load into a variable 'ser' the information about the usb you are listening. /dev/tty.usbserial.... is the port after plugging in the hygrometer, 9600 is for bauds, it can be diminished
count = 0
while 1:
    read_byte = ser.read(size=1)

So now I want to find the end of the line of the data as the measurement informations that I need are in a line that begins with 'V', and if the data sheet of my sensor, it said that a line ends by , so I want to read one byte at a time and look for '<', then 'c', then 'r', then '>'. So I wanted to do this:

while 1:
    read_byte = ser.read(size=8) #read a byte
    read_byte_hexa =binascii.hexlify(read_byte) #convert the byte into hexadecimal

    trad_hexa = int(read_byte_hexa , 16) #convert the hexadecimal into an int in purpose to compare it with another int
    trad_firstcrchar = int('3c' , 16) #convert the hexadecimal of the '<' into a int to compare it with the first byte    
    if (trad_hexa == trad_firstcrchar ): #compare the first byte with the '<'    
        read_byte = ser.read(size=1) #read the next byte (I am not sure if that really works)
        read_byte_hexa =binascii.hexlify(read_byte)# from now I am doing the same thing as before
        trad_hexa = int(read_byte_hexa , 16)
        trad_scdcrchar = int('63' , 16)
        print(trad_hexa, end='/')# this just show me if it gets in the condition
        print(trad_scdcrchar)    
        if (trad_hexa == trad_scdcrchar ):    
            read_byte = ser.read(size=1) #read the next byte 
            read_byte_hexa =binascii.hexlify(read_byte)
            trad_hexa = int(read_byte_hexa , 16)
            trad_thirdcrchar = int('72' , 16)
            print(trad_hexa, end='///')
            print(trad_thirdcrchar)    
            if (trad_hexa == trad_thirdcrchar ):    
                read_byte = ser.read(size=1) #read the next byte 
                read_byte_hexa =binascii.hexlify(read_byte)
                trad_hexa = int(read_byte_hexa , 16)
                trad_fourthcrchar = int('3e' , 16)
                print(trad_hexa, end='////')
                print(trad_fourthcrchar)    
                if (trad_hexa == trad_fourthcrchar ):    
                    print ('end of the line')

But I am not sure that it works, I mean I think it does not have the time to read the second one, the second byte I am reading, it's not exactly the second one. So that's why I want to use a buffer, but I don't really get how I can do that. I am going to look for it, but if someone knows an easier way to do what I want, I am ready to try it! Thank you

Answer 1

You seem to be under the impression that the end-of-line character for that sensor's communication protocol is 4 different characters: <, c, r and >. However, what is being referred to is the carriage return, often denoted by <cr> and in many programming languages just by \r (even though it looks like 2 characters, it represents just one character).

You could simplify your code greatly by reading in the data from the sensors line by line, as the protocol is structured. Here's something to help you get started:

import time

def parse_info_line(line):
    # implement to your own liking
    logical_channel, physical_probe, hardware_id, crc = [line[index:index+2] for index in (1, 3, 5, 19)]
    serialno = line[7:19]
    return physical_probe

def parse_value_line(line):
    channel, crc = [line[ind:ind+2] for ind in (1,7)]
    encoded_temp = line[3:7]
    return twos_comp(int(encoded_temp, 16), 16)/100.

def twos_comp(val, bits):
    """compute the 2's compliment of int value `val`"""
    if (val & (1 << (bits - 1))) != 0: # if sign bit is set e.g., 8bit: 128-255
        val = val - (1 << bits)        # compute negative value
    return val                         # return positive value as is

def listen_on_serial(ser):
    ser.readline() # do nothing with the first line: you have no idea when you start listening to the data broadcast from the sensor
    while True:
        line = ser.readline()
        try:
            first_char = line[0]
        except IndexError:  # got no data from sensor
            break
        else:
            if first_char == '@':  # begins a new sensor record
                in_record = True
            elif first_char == '$':
                in_record = False
            elif first_char == 'I':
                parse_info_line(line)
            elif first_char == 'V':
                print(parse_value_line(line))
            else:
                print("Unexpected character at the start of the line:\n{}".format(line))
            time.sleep(2)

The twos_comp function was written by travc and you are encouraged to upvote his answer when you have enough reputation and if you intend to use his code (and even if you won't, it's still a good answer, I upvoted it just now). The listen_on_serial could be improved as well (many Python programmers will recognize the switch-structure and implement it with a dictionary rather than if... elif... elif...), but this is only intended to get you started.

As a test, the following code extract simulates the sensor sending some data (which is line-delimited, using the carriage return as the end-of-line marker), which I copied from the pdf you linked to (FAQ_terminalfenster_E.pdf).

>>> import serial
>>> import io
>>> 
>>> ser = serial.serial_for_url('loop://', timeout=1)
>>> serio = io.TextIOWrapper(io.BufferedRWPair(ser, ser), newline='\r', line_buffering=True)
>>> serio.write(u'A1A0\r'  # simulation of starting to listen halfway between 2 records
...     '$\r'              # marks the end of the previous record
...     '@\r'              # marks the start of a new sensor record
...     'I0101010000000000001B\r'  # info about a sensor's probe
...     'V0109470D\r'              # data matching that probe
...     'I0202010000000000002B\r'  # other probe, same sensor
...     'V021BB55C\r')             # data corresponding with 2nd probe
73L
>>> 
>>> listen_on_serial(serio)
23.75
70.93
>>> 

Note that it is recommended by the pyserial docs to be using TextIOWrapper when the end-of-line character is not \n (the linefeed character), as was also answered here.