3

I am writing a program that needs to serialize small (up to 16 bit) signed integers to a binary file. As part of that, I need to write a function that can split an integer into two bytes and another function that can convert this pair of bytes back to the original integer.

The first idea that came to my mind was to solve this problem similarly to how I would solve it C:

function dump_i16(n)
  assert (-0x8000 <= n and n < 0x8000) -- 16 bit
  local b1 = (n >> 8) & 0xff
  local b2 = (n >> 0) & 0xff
  return b1, b2
end

function read_i16(b1, b2)
  assert (0 <= b1 and b1 <= 0xff) -- 8 bit
  assert (0 <= b2 and b2 <= 0xff) -- 8 bit
  return (b1 << 8) | (b2 << 0)
end

However, these functions break on negative numbers because Lua integers actually have 64 bits and I am only preserving the lower 16 bits:

-- Positive numbers are OK
print( read_i16(dump_i16(17)) ) -- 17
print( read_i16(dump_i16(42)) ) -- 42

-- Negative numbers don't round trip.
print( read_i16(dump_i16(-1)) )  -- 65535 = 2^16 - 1
print( read_i16(dump_i16(-20)) ) -- 65516 = 2^16 - 20

What would be the cleanest way to modify my read_i16 function so it works correctly for negative numbers?

I would prefer to do this using pure Lua 5.3 if possible, without going down to writing C code.

hugomg
  • 68,213
  • 24
  • 160
  • 246
  • Your dump_i16 is producing unsigned values [0,256), so it is always going to fail your assertions – Moop Dec 03 '16 at 05:31
  • You said that `read_i16(dump_i16(-20) == 65516` and `65516 == 2^16 -20`. Why don't you just subtract `2^16` from the result ...? – siffiejoe Dec 03 '16 at 06:38
  • @Moop: You are right. I should have tested before editing those in in the middle of the night... – hugomg Dec 03 '16 at 12:46

3 Answers3

4

Lua 5.3 has special conversion functions.

To convert an integer -32768..32767 to string of length 2 in big-endian order: 

local a_string = string.pack(">i2", your_integer)

To convert it back (first two bytes of "a_string" are being converted):

local an_integer = string.unpack(">i2", a_string)
hugomg
  • 68,213
  • 24
  • 160
  • 246
Egor Skriptunoff
  • 23,359
  • 2
  • 34
  • 64
1

You need a logical right shift for dealing with the sign. See Difference between >>> and >>. Lua doesn't have it, so you have do your own. Since Lua integers are 64 by default, you mask off 64 minus the number of shifts you do.

function dump_i16(n)
   assert (-0x8000 <= n and n < 0x8000)
   local b1 = (n >> 8) & ~(-1<<(64-8))
   local b2 = (n >> 0) & 0xff
   return b1, b2
end
Community
  • 1
  • 1
Moop
  • 3,414
  • 2
  • 23
  • 37
  • Is there a way to modify `read_i16` instead of `dump_i16`? I need b1 and b2 to both fit in a single byte. – hugomg Dec 03 '16 at 05:00
  • 1
    Lua 5.2 doesn't support `<<` and `>>`-style bitwise operations. Could you add a solution for Lua 5.2? – avs Oct 18 '21 at 07:47
0

One way to fix read_i16 is to use the sign extension algorithm that is used internally by string.pack:

function read_i16(b1, b2)
  assert (0 <= b1 and b1 <= 0xff)
  assert (0 <= b2 and b2 <= 0xff)
  local mask = (1 << 15)
  local res  = (b1 << 8) | (b2 << 0)
  return (res ~ mask) - mask
end
hugomg
  • 68,213
  • 24
  • 160
  • 246