Decode Payload of TCP connection: 'utf-8' codec can't decode byte 0x87 in position 4: invalid start byte

Question

I am sending from a sensor node over TCP to my TCP server. The raw received data looks like:

b'A\x10Vu\x87%\x00x\x0c\xc7\x03\x01\x00\x00\x00\x00&\x00\x00\x00\x00\x00\x00\x00\x00'

When trying to decode it using utf-8, I receive the following error. Code:

my_variable = b'A\x10Vu\x87%\x00x\x0c\xc7\x03\x01\x00\x00\x00\x00&\x00\x00\x00\x00\x00\x00\x00\x00'

print(my_variable.decode('utf-8'))

Error:

print(my_variable.decode('utf-8')) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 4: invalid start byte

So the problem is that the Payload contains non-ascii format characters, apparently.

How can I decode this payload to sth. human readable?

The payload description can be found here on p32. p20 shows a tcp connection example but without decoding the payload.

Do you need to decode it? If it's raw sensor data, it may not be proper to try and represent it as a decoded string. — Carcigenicate, Jul 09 '21 at 14:29
in documentation at the top of page 32 you can see `"The payload is ASCII string"` so you should't decode it with `utf-8` but `ascii`. But probably you shouldn't even decode it but create special code which read bytes and convert to integer values. — furas, Jul 09 '21 at 14:30
first: what informations did you send from sensor? It doesn't send it as `human readable` and it may need to split it to smaller parts and convert every part from hex to integer - so it may need much complex code. — furas, Jul 09 '21 at 14:39
@furas thank you for mentioning ascii, I also receive an error for decoding to ascii: UnicodeDecodeError: 'ascii' codec can't decode byte 0x87 in position 4: ordinal not in range(128) The payload is described quite well in the documentation sheet and I have no problems using MQTT. Here is an ascii example from the MQTT test 41105675872500780cc0030100000000270000000000000000 How can I convert my tcp payload to sth. like this? — Marco Bob, Jul 09 '21 at 14:45
Possible duplicate of https://stackoverflow.com/questions/62170614/python-unicodedecodeerror-utf-8-codec-cant-decode-byte-0x80-in-position-0 — tripleee, Jul 09 '21 at 16:04
In case you still need it, i have written a decoder for the Dragino NBSN95 devices that can handle all payload types. It can be found here and is MIT licensed, https://gist.github.com/laundmo/b807f7b804242f2407cceb08120924bb — laundmo, Dec 09 '21 at 12:28

furas · Accepted Answer · 2021-07-09T15:16:19.273

Base on documentation is is NOT human readable and you shouldn't decode it but you should write special code to convert every value from hex to integer and eventually convert it to string with extra values - ie. dots in version number.

Here code for beginning values from payload

data = b'A\x10Vu\x87%\x00x\x0c\xc7\x03\x01\x00\x00\x00\x00&\x00\x00\x00\x00\x00\x00\x00\x00'

ID = data[:6].hex()
print('ID:', ID)

hardware = data[6]
if hardware == 1:
    hardware = 'NBSN95'
print('hardware version:', hardware)

software = data[7]
print('software version (raw):', software)

software = '.'.join(list(str(software)))
print('software version:', software)

battery = data[8:10].hex()
print('battery (raw):', battery)

battery = int(battery, 16)
print('battery:', battery, 'mV =', battery/1000, 'V')

signal = data[10]
print('signal (raw):', signal)    

if signal == 0:
    signal = '-113dBm or less'
elif signal == 1:
    signal = '-111dBm'
elif 2 <= signal <= 30:
    signal = '-109dBm ... -53dBm'
elif signal == 31:     
    signal = '-51dBm or greater'
elif signal == 99:
    signla = 'Not known or not detectable'
print('signal:', signal)    
    
temp = data[11:13].hex()    
print('temperature (raw):', temp)

temp = int(temp, 16)
if temp & 0xFC00 == 0:
    temp = temp/10
elif temp & 0xFC00 == 1:
    temp = (temp-65536)/10
print('temperature:', temp, 'degree')

Result:

ID: 411056758725
hardware version: 0
software version (raw): 120
software version: 1.2.0
battery (raw): 0cc7
battery: 3271 mV = 3.271 V
signal (raw): 3
signal: -109dBm ... -53dBm
temperature (raw): 0100
temperature: 25.6 degree

score 0 · Answer 2 · answered Jul 09 '21 at 15:34

You could find the answer in a Python prompt. In fact, I started my exploration using dir(my_variable):

my_variable = b'A\x10Vu\x87%\x00x\x0c\xc7\x03\x01\x00\x00\x00\x00&\x00\x00\x00\x00\x00\x00\x00\x00'
# dir(my_variable)
my_variable.hex

<built-in method hex of bytes object at 0x00000232BDF129F0>

help(my_variable.hex)            # truncated

Help on built-in function hex:

hex(...) method of builtins.bytes instance
    Create a str of hexadecimal numbers from a bytes object.

my_variable.hex()

'41105675872500780cc7030100000000260000000000000000'

Mark Tolonen · Answer 3 · 2021-07-13T18:48:18.803

The data is raw byte data and should not be decoded. Instead, use the struct module to unpack the raw bytes into bytes and words of data. The spec (page 22) indicates how many bytes for each field:

packet layout

The struct module also has the advantage that you don't have to manually calculate the offsets to each fields, and if the unpack pattern doesn't match the length of data it will catch the error.

Note that the 2-byte version is a hardware version byte and a software version byte, so I used BB (2 bytes) to extract them separately. The temperatures were documented as two's complement so I used h (signed 16-bit value) for them. Also note the data is big-endian, so > is used.

See also the struct - Format String documentation.

import struct
from datetime import datetime
from pytz import UTC

data = b'A\x10Vu\x87%\x00x\x0c\xc7\x03\x01\x00\x00\x00\x00&\x00\x00\x00\x00\x00\x00\x00\x00'

devid,hw,sw,bat,ss,mod,t1,dii,adc,t2,h,ts = struct.unpack('>6sBBHBBhBHhHL',data)

# fields that needed processing are done in the f-strings below
print(f"DeviceID={devid.hex()} HW={hw} SW={'.'.join(str(sw))}\n"
      f"BAT={bat:.3f}mV SignalStrength={-113+2*ss}dBm Mode={mod} Temp={t1/10}\N{DEGREE CELSIUS}\n"
      f"Door={dii==0x80} ADC={adc}mv Temp2={t1/10:.1f}\N{DEGREE CELSIUS} Humidity={h/10:.1f}%\n"
      f"Timestamp={datetime.fromtimestamp(ts,UTC)}")

Output:

DeviceID=411056758725 HW=0 SW=1.2.0
BAT=3.271V SignalStrength=-107dBm Mode=1 Temp=0.0℃
Door=False ADC=38mv Temp2=0.0℃ Humidity=0.0%
Timestamp=1970-01-01 00:00:00+00:00

Decode Payload of TCP connection: 'utf-8' codec can't decode byte 0x87 in position 4: invalid start byte

3 Answers3