Unusual Integer encoding to bytes - what scheme is this?

Question

What Integer encoding scheme is used to achieve the following, and how can we do this in .Net:

Are you sure this is encoding? Looks like someone was rearranging bytes in dwords and not getting it quite right. — GSerg, Jul 19 '15 at 13:41
@GSerg the hex dumps from the data source are consistently in this pattern, hence i feel it is a form of encoding — Charles Okwuagwu, Jul 19 '15 at 13:43
@GSerg this is pattern that was used to encode the length of the binary payload that followed in each case. My manual length calculation was accurate, hence i see this as some integer (length) encoding pattern — Charles Okwuagwu, Jul 19 '15 at 13:47

Hans Passant · Accepted Answer · 2015-07-19T13:54:28.190

10

Not so sure it has an official name, it is a 7-bit encoding. It is a variable length encoding, the high bit of a byte is set if another byte follows. Byte order is little-endian.

The .NET Framework uses it, Write7BitEncodedInt() method. Used by the BinaryWriter.WriteString() method, it saves space since most practical strings have less than 128 characters.

So F403 => 03F4 => |0000011|1110100| => |00000001|11110100| => 0x1F4 == 500

edited Jul 19 '15 at 13:54

answered Jul 19 '15 at 13:51

Hans Passant

922,412
146
1,693
2,536

can you kindly provide an example please, in particular, convert 128 and 256. Why does it lead with 80? – Charles Okwuagwu Jul 19 '15 at 13:53
1

I added a non-trivial example. You see 80 simply because the high bit is set, it encodes 7 bits of 0. If you need code then just copy/paste the framework source. The decoder [is here](http://referencesource.microsoft.com/#mscorlib/system/io/binaryreader.cs,f30b8b6e8ca06e0f). – Hans Passant Jul 19 '15 at 13:55
Just seen your example. Do we have to handle this manually then, in .net to get the expected output with leading 80 in some cases? – Charles Okwuagwu Jul 19 '15 at 13:58
The link you provided is for the BinaryReader.cs I have looked at the binaryWriter.cs http://referencesource.microsoft.com/#mscorlib/system/io/binarywriter.cs,0dcb1e4b7b6cce02 . The equivalent VB.net fails when you try to do cbyte(value), for value >255. – Charles Okwuagwu Jul 19 '15 at 14:48
Sure, that is not valid in VB.NET, only in C# and only if overflow checking is disabled. You'll have to use CByte(value And &HFF). Click the Ask Question button if you have more questions about this. – Hans Passant Jul 19 '15 at 14:59
the solution example you provided works in one direction. Please kindly provide the reverse: how do we go from 500 ==> F403. See my attempt in the edited question above. Thanks. – Charles Okwuagwu Jul 19 '15 at 15:24
I seriously doubt that will help, your code is not remotely close. If you don't want to ask a question about it then google "online c# to vb.net translator". The Telerik hit does a decent job of it, use MemoryStream.Write(). – Hans Passant Jul 19 '15 at 15:30
Thanks Hans, i've resolved it. These c# to vb.net converters don't always get it right, for example, they won't give CByte(value And &HFF) for values >=256. I feel it's best to understand the problem, then re-write the code. – Charles Okwuagwu Jul 20 '15 at 18:06
I don't understand the example. For me it should be: F4 03 => ‭1000_0011 1110_1000 0000_0011‬ => 83 E8 03. To retrieve the original value, you start from the right and you remove the high bit: 000_0011‬ 110_1000 000_0011 => 11‬11_0100 0000_0011 – Maxence Jan 13 '20 at 17:38

score 1 · Answer 2 · answered Jul 19 '15 at 17:19

SOLVED. I Hope this helps someone else.

    Dim o = {127, 128, 255, 256, 500}

    For Each i As Integer In o
        Console.WriteLine("{0} = {1}", i, Write(i))
    Next

Function Write(value As Short) As String
    Dim a = New List(Of Byte)

    ' Write out an int 7 bits at a time.  The high bit of the byte, 
    ' when on, tells reader to continue reading more bytes.
    Dim v = CShort(value)
    ' support negative numbers
    While v >= &H80
        a.Add(CByte((v And &HFF) Or &H80))
        v >>= 7
    End While

    a.Add(CByte((v And &HFF)))

    Return B2H(a.ToArray)
End Function

Function B2H(b() As Byte) As String
    Return BitConverter.ToString(b).Replace("-", "")
End Function

Result:

Unusual Integer encoding to bytes - what scheme is this?

2 Answers2

Linked