Encode/decode data from RS232 serial port

Question

This is the first time I've had to connect to a device via RS232 serial to read/write data and I'm stuck on the encoding/decoding procedures.

I'm doing everything in Python 3 using the library "pyserial". Here is what I've done so far:

import serial

ser = serial.Serial()
ser.port = '/dev/ttyUSB0'
ser.baudrate = 115200
ser.bytesize = serial.EIGHTBITS
ser.parity = serial.PARITY_NONE
ser.stopbits = serial.STOPBITS_ONE
ser.timeout = 3

ser.open()

device_write = ser.write(bytearray.fromhex('AA 55 00 00 07 00 12 19 00'))

device_read = ser.read_until()

The connection/communication appears to be working as intended. The output of device_read is

b'M1830130A2IMU v3.2.9.1 26.04.19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x0527641\x00\x00\x00IMHF R.1.0.0 10.28.2018 td:  6.500ms\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00'

and this is where I'm stuck. I don't know how to interpret this. Attached is an image from the datasheet which explains what the output is suppose to represent.

The datasheet says "fields in bytes 98 to 164 are empty" for the device I have. Can someone help me understand what needs to be done to convert the output of ser.read_until() to a form that is "human readable" and represents the data in the image? I don't need someone to write the code for me, but I'm not even sure where to start. Again, this is my first time doing this so I'm a bit lost on what is going on.

The data being read doesn't look right, so it seems likely that there's a communications problem. — martineau, Mar 24 '20 at 01:02
Ahh...you're right. The "command" called for writing 9 bytes when I was only sending 1 (I didn't include the header, check sum, etc. bytes). I updated the output — ThatsRightJack, Mar 25 '20 at 05:10
Good to hear — so, do you now know how to decode the information? — martineau, Mar 25 '20 at 12:07
If I do `device_read[0:8].decode('ascii')`, I get `'M1830130'`. If I do `device_read[8:48].decode('ascii')` I get `'A2IMU v3.2.9.1 26.04.19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'`. I don't see how this "decoded" to a "human readable" form? In other words, why do I still have things like `\x00` in the output? Also the output of `device.read[48]` is equal to (b'\x02') which gives 2 as opposed to a 1 or 0 as defined in the datasheet. This makes be believe either I'm still doing something wrong, or the datasheet is incorrect (less likely). — ThatsRightJack, Mar 25 '20 at 22:30
\x00 is just 0, which means null, which is the standard end marker for a string whose length you don't know. You can strip it like so: ``` ID_fw = device_read[8:48].decode('ascii') first_zero = ID_fw.find('\x00') if first_zero >= 0: ID_fw = ID_fw[0:first_zero] ``` — Ozan Bellik, Mar 25 '20 at 23:59
A simple way to remove the trailing NUL bytes would be e.g. `ID_fw = device_read[8:48].decode('ascii').rstrip('\x00')`. It *is* strange that the `Press_sens` value is 2, not 1 or 0. Any way to contact the hardware vendor? — martineau, Mar 26 '20 at 02:34

Ozan Bellik · Accepted Answer · 2020-03-24T00:29:06.770

4

If you are trying to write a single byte with hex value 12 (decimal 18), I believe what you need to do is ser.write(bytes([0x12])), which is equivalent to ser.write(bytes([18])).

It looks like your output is 154 bytes rather than 98, and much of it non-human-readable. But if you did have the data described in the graph, you could break it up like this:

ID_sn = device_read[0:8].decode('ascii')
ID_fw = device_read[8:48].decode('ascii')
Press_Sens = device_read[48]

and so on.

edited Mar 24 '20 at 00:29

answered Mar 24 '20 at 00:22

Ozan Bellik

483
2
6

I typed in `device_read[0:8]` and it returned the following byte sequence `b'\xf3\x02\x00\x13\xda\xf1\xff\x00'`. I then appended `decode('ascii')` to the command (e.g. `device_read[0:8].decode('ascii')` and I got an error which said `UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 0: ordinal not in range(128)`. Can you comment on what this means? – ThatsRightJack Mar 24 '20 at 01:10
A byte gives you a range of values between 0 and 255. There are many ways to map those values to meaning (e.g. characters). ASCII is one of those ways and the de facto standard for representation that use one and only one byte per character (which isn't how we do it with new systems nowadays). So I guessed RS232 uses ASCII. ASCII only uses the values from 0 to 127 (and leaves the rest for custom uses). 0xf3 is 243 -- outside that range, so not valid ASCII (but could be part of a number). Other potential encodings would be UTF-8 and UTF-16, but you'll see those don't work, either. – Ozan Bellik Mar 24 '20 at 01:21
So, I'm pretty sure what you're getting is not what's described in the table. Otherwise you'd actually be able to read all but two bytes of the 98 (and you have more than that many bytes) by just looking at the b' string. Have you tried passing in bytes([0x12]) instead of 0x12? – Ozan Bellik Mar 24 '20 at 01:23
edit: not RS232 but your device based on your posted snippet of the datasheet. – Ozan Bellik Mar 24 '20 at 01:42
The command I was writing wasn't correct. I updated it, as well as the output. I also updated the datasheet snippet. If I type in `len(device_read)` it outputs 166, which I believe is correct from the datasheet. Do I have to use the "Format" column for the decoding process? – ThatsRightJack Mar 25 '20 at 05:23
Yes, Format tells you how to interpret the bytes. – Ozan Bellik Mar 26 '20 at 00:00
The `word` might be tricky -- it's a multi-byte number. You have to know the endianness -- whether the first byte the most or the least significant. And then you have to combine accordingly (either `device_read[162]*256+device_read[163]` or vice versa). – Ozan Bellik Mar 26 '20 at 00:11
@ozangds: No need to worry about the endianess of the `word` because the OP wrote: "..."fields in bytes 98 to 164 are empty for the device I have". – martineau Mar 26 '20 at 03:01
@ozangds I do have other stuff I have to decode in which `word` does come into play. In the datasheet it says "word = unsigned 2 byte integer;" and it also says "The low byte of all data defined as word, sword, float is transmitted first." Can you elaborate a bit on the example you gave a few comments up? Where did the 256 come from? Using your example, how does this relate to the info I was given in the datasheet? – ThatsRightJack Mar 30 '20 at 01:30
1

@ThatsRightJack -- the two bytes are basically like a 16-digit number except that the digits are only 0-1 instead of 0-9. If you had a 16-digit number and you were reading it 8 digits at a time, you'd multiply the first 8-digit part (the higher order part) by 100 million (1 with 8 zeros) before adding it to the other one, right? In binary, 1 with 8 zeros after it is the number 256 (2 to the eighth). And what the data sheet tells you is that the second byte is the higher order one, so you need to multiply the second one by 256 before adding it to the first. – Ozan Bellik Mar 30 '20 at 18:27

score 2 · Answer 2 · edited Mar 30 '20 at 01:36

This isn't an answer, just @ozangds' idea fleshed-out (might save you some typing):

def decode_bytes(data, start, stop):
    return data[start:stop+1].decode('ascii').rstrip('\x00')


device_read = b'M1830130A2IMU v3.2.9.1 26.04.19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x0527641\x00\x00\x00IMHF R.1.0.0 10.28.2018 td:  6.500ms\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00'

ID_sn = decode_bytes(device_read, 0, 7)
ID_fw = decode_bytes(device_read, 8, 47)
Press_sens = device_read[48]
IMU_type = device_read[49]
IMU_sn = decode_bytes(device_read, 50, 57)
IMU_fw = decode_bytes(device_read, 58, 97)

label_fmt = '{:>10}: {!r}'
print(label_fmt.format('ID_sn', ID_sn))
print(label_fmt.format('ID_fw', ID_fw))
print(label_fmt.format('Press_sens', Press_sens))
print(label_fmt.format('IMU_type', IMU_type))
print(label_fmt.format('IMU_sn', IMU_sn))
print(label_fmt.format('IMU_fw', IMU_fw))

Output:

     ID_sn: 'M1830130'
     ID_fw: 'A2IMU v3.2.9.1 26.04.19'
Press_sens: 2
  IMU_type: 5
    IMU_sn: '27641'
    IMU_fw: 'IMHF R.1.0.0 10.28.2018 td:  6.500ms'

Encode/decode data from RS232 serial port

2 Answers2

Linked