Getting wrong values when I stitch 2 shorts back into an unsigned long

Question

I am doing BLE communications with an Arduino Board and an FPGA. I have this requirement which restraints me from changing the packet structure (the packet structure is basically short data types). Thus, to send a timestamp (form millis()) over, I have to split an unsigned long into 2 shorts on the Arduino side and stitch it back up on the FPGA side (python).

This is the implementation which I have:

  // Arduino code in c++
  unsigned long t = millis();
  // bitmask to get bits 1-16
  short LSB = (short) (t & 0x0000FFFF); 
  // bitshift to get bits 17-32
  short MSB = (short) (t >> 16);

  // I then send the packet with MSB and LSB values

  # FPGA python code to stitch it back up (I receive the packet and extract the MSB and LSB)
  MSB = data[3]
  LSB = data[4]
  data = MSB << 16 | LSB

Now the issue is that my output for data on the FPGA side is sometimes negative, which tells me that I must have missed something somewhere as timestamps are not negative. Does any one know why ?

When I transfer other data in the packet (i.e. other short values and not the timestamp), I am able to receive them as expected, so the problem most probably lies in the conversion that I did and not the sending/receiving of data.

Does this answer your question https://stackoverflow.com/questions/20766813/how-to-convert-signed-to-unsigned-integer-in-python ? — Slava, Feb 20 '20 at 15:21
What values are in `data`? What are the types? Mapping the bits into a type that can be negative while on the C++ side they are unsigned is an error in and of itself. BTW: You should tag this with "verilog" or "vhdl", assuming that's the language you use to program the FPGA. — Ulrich Eckhardt, Feb 20 '20 at 15:22

guidot · Accepted Answer · 2020-02-20T15:56:29.950

2

short defaults to signed, and in case of a negative number >> will keep the sign by shifting in one bits in from the left. See e.g. Microsoft.

From my earlier comment: In Python avoid attempting that by yourself (by the way short from C perspective has no idea concerning its size, you always have to look into the compiler manual or limits.h) and use the struct module instead.

edited Feb 20 '20 at 15:56

answered Feb 20 '20 at 15:50

guidot

5,095
2
25
37

score 0 · Answer 2 · answered Feb 20 '20 at 15:13

0

you probably need/want to first convert the long to network byte order using hotnl

answered Feb 20 '20 at 15:13

nivpeled

1,810
5
17

Val · Answer 3 · 2020-03-10T22:10:15.503

As guidot reminded “short” is signed and as data are transferred to Python the code has an issue:

For t=0x00018000 most significant short MSB = 1, least significant short LSB = -32768 (0x8000 in C++ and -0x8000 in Python) and Python code expression
time = MSB << 16 | LSB
returns time = -32768 (see the start of Python code below).
So, we have incorrect sign and we are loosing MSB (any value, not only 1 in our example).
MSB is lost because in the expression above LSB is extended with sign bit 1 to the left 16 bits, then new 16 “1” bits override with “|” operator whatever MSB we have and then all new 16 “1” bits are discarded and the expression returns LSB.

Straightforward fix (1.1 Fix) would be fixing MSB, LSB to unsigned short. This could be enough without any changes in Python code.

To exclude bit operations we could use “union” as per 1.2 Fix.

Without access to C++ code we could fix in Python by converting signed LSB, MSB (2.1 Fix) or use “Union” (similar to C++ “union”, 2.2 Fix).

C++

#include <iostream>
using namespace std;

int main () {
  unsigned long t = 0x00018000;
  short LSB = (short)(t & 0x0000FFFF);
  short MSB = (short)(t >> 16);
  cout << hex << "t = " << t << endl;
  cout << dec << "LSB = " << LSB << " MSB = " << MSB << endl;

  // 1.1 Fix Use unsigned short instead of short
  unsigned short fixedLSB = (unsigned short)(t & 0x0000FFFF);
  unsigned short fixedMSB = (unsigned short)(t >> 16);
  cout << "fixedLSB = " << fixedLSB << " fixedMSB = " << fixedMSB << endl;

  // 1.2 Fix Use union
  union {
    unsigned long t2;
    unsigned short unsignedShortArray[2];
  };

  t2 = 0x00018000;

  fixedLSB = unsignedShortArray [0];
  fixedMSB = unsignedShortArray [1];

  cout << "fixedLSB = " << fixedLSB << " fixedMSB = " << fixedMSB << endl;
}

Output

t = 18000
LSB = -32768 MSB = 1
fixedLSB = 32768 fixedMSB = 1
fixedLSB = 32768 fixedMSB = 1

Python

DATA=[0, 0, 0, 1, -32768]

MSB=DATA[3]
LSB=DATA[4]
data = MSB << 16 | LSB
print (f"MSB = {MSB} ({hex(MSB)})")
print (f"LSB = {LSB} ({hex(LSB)})")
print (f"data = {data} ({hex(data)})")

time = MSB << 16 | LSB
print (f"time = {time} ({hex(time)})")


# 2.1 Fix
def twosComplement (short):
  if short >= 0: return short
  return 0x10000 + short
fixedTime = twosComplement(MSB) << 16 | twosComplement(LSB)

# 2.2 Fix
import ctypes
class UnsignedIntUnion(ctypes.Union):
    _fields_ = [('unsignedInt', ctypes.c_uint),
                ('ushortArray', ctypes.c_ushort * 2),
                ('shortArray', ctypes.c_short * 2)]

unsignedIntUnion = UnsignedIntUnion(shortArray = (LSB, MSB))

print ("unsignedIntUnion")
print ("unsignedInt = ", hex(unsignedIntUnion.unsignedInt))
print ("ushortArray[1] = ", hex(unsignedIntUnion.ushortArray[1]))
print ("ushortArray[0] = ", hex(unsignedIntUnion.ushortArray[0]))

print ("shortArray[1] = ", hex(unsignedIntUnion.shortArray[1]))
print ("shortArray[0] = ", hex(unsignedIntUnion.shortArray[0]))

unsignedIntUnion.unsignedInt=twosComplement(unsignedIntUnion.shortArray[1]) << 16 | twosComplement(unsignedIntUnion.shortArray[0])

def toUInt(msShort: int, lsShort: int):
  return UnsignedIntUnion(ushortArray = (lsShort, msShort)).unsignedInt

fixedTime = toUInt(MSB, LSB)
print ("fixedTime = ", hex(fixedTime))

print()

Output

MSB = 1 (0x1)
LSB = -32768 (-0x8000)
data = -32768 (-0x8000)
time = -32768 (-0x8000)
unsignedIntUnion
unsignedInt =  0x18000
ushortArray[1] =  0x1
ushortArray[0] =  0x8000
shortArray[1] =  0x1
shortArray[0] =  -0x8000
fixedTime =  0x18000

Getting wrong values when I stitch 2 shorts back into an unsigned long

3 Answers3