hpack encoding integer significance

Question

After reading this, https://httpwg.org/specs/rfc7541.html#integer.representation I am confused about quite a few things, although I seem to have the overall gist of the idea. For one, What are the 'prefixes' exactly/what is their purpose?
For two:

C.1.1. Example 1: Encoding 10 Using a 5-Bit Prefix

The value 10 is to be encoded with a 5-bit prefix.

    10 is less than 31 (2^5 - 1) and is represented using the 5-bit prefix.

  0   1   2   3   4   5   6   7
+---+---+---+---+---+---+---+---+
| X | X | X | 0 | 1 | 0 | 1 | 0 |   10 stored on 5 bits
+---+---+---+---+---+---+---+---+

What are the leading Xs? What is the starting 0 for?

>>> bin(10)
'0b1010'
>>> 

Typing this in the python IDE, you see almost the same output... Why does it differ? This is when the number fits within the number of prefix bits though, making it seemingly simple.

C.1.2. Example 2: Encoding 1337 Using a 5-Bit Prefix

The value I=1337 is to be encoded with a 5-bit prefix.

    1337 is greater than 31 (25 - 1).
        The 5-bit prefix is filled with its max value (31).
    I = 1337 - (25 - 1) = 1306.
        I (1306) is greater than or equal to 128, so the while loop body executes:
            I % 128 == 26
            26 + 128 == 154
            154 is encoded in 8 bits as: 10011010
            I is set to 10 (1306 / 128 == 10)
            I is no longer greater than or equal to 128, so the while loop terminates.
        I, now 10, is encoded in 8 bits as: 00001010.
    The process ends.

  0   1   2   3   4   5   6   7
+---+---+---+---+---+---+---+---+
| X | X | X | 1 | 1 | 1 | 1 | 1 |  Prefix = 31, I = 1306
| 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 |  1306>=128, encode(154), I=1306/128
| 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 |  10<128, encode(10), done
+---+---+---+---+---+---+---+---+

The octet-like diagram shows three different numbers being produced... Since the numbers are produced throughout the loop, how do you replicate this octet-like diagram within an integer? What is the actual final result? The diagram or "I" being 10, or 00001010.

def f(a, b):
    if a < 2**b - 1:
        print(a)
    else:
        c = 2**b - 1
        remain = a - c
        print(c)
        if remain >= 128:
            while 1:
                e = remain % 128
                g = e + 128
                remain = remain / 128
                if remain >= 128:
                    continue
                else:
                    print(remain)
                    c+=int(remain)
                    print(c)
                    break

As im trying to figure this out, I wrote a quick python implementation of it, It seems that i am left with a few useless variables, one being g which in the documentation is the 26 + 128 == 154.
Lastly, where does 128 come from? I can't find any relation between the numbers besides the fact 2 raised to the 7th power is 128, but why is that significant? Is this because the first bit is reserved as a continuation flag? and an octet contains 8 bits so 8 - 1 = 7?

Answer 1

For one, What are the 'prefixes' exactly/what is their purpose?

Integers are used in a few places in HPACK messages and often they have leading bits that cannot be used to for the actual integer. Therefore, there will often be a few leading digits that will be unavailable to use for the integer itself. They are represented by the X. For the purposes of this calculation it doesn't make what those Xs are: could be 000, or 111, or 010 or...etc. Also, there will not always be 3 Xs - that is just an example. There could only be one leading X, or two, or four...etc.

For example, to look up a previous HPACK decoded header, we use 6.1. Indexed Header Field Representation which starts with a leading 1, followed by the table index value. Therefore that 1 is the X in the previous example. We have 7-bits (instead of only 5-bits in the original example in your question). If the table index value is 127 or less we can represent it using those 7-bits. If it's >= 127 then we need to do some extra work (we'll come back to this).

If it's a new value we want to add to the table (to reuse in future requests), but we already have that header name in the table (so it's just a new value for that name we want as a new entry) then we use 6.2.1. Literal Header Field with Incremental Indexing. This has 2 bits at the beginning (01 - which are the Xs), and we only have 6-bits this time to represent the index of the name we want to reuse. So in this case there are two Xs.

So don't worry about there being 3 Xs - that's just an example. In the above examples there was one X (as first bit had to be 1), and two Xs (as first two bits had to be 01) respectively. The Integer Representation section is telling you how to handle any prefixed integer, whether prefixed by 1, 2, 3... etc unusable "X" bits.

What are the leading Xs? What is the starting 0 for?

The leading Xs are discussed above. The starting 0 is just because, in this example we have 5-bits to represent the integers and only need 4-bits. So we pad it with 0. If the value to encode was 20 it would be 10100. If the value was 40, we couldn't fit it in 5-bits so need to do something else.

Typing this in the python IDE, you see almost the same output... Why does it differ?

Python uses 0b to show it's a binary number. It doesn't bother showing any leading zeros. So 0b1010 is the same as 0b01010 and also the same as 0b00001010.

This is when the number fits within the number of prefix bits though, making it seemingly simple.

Exactly. If you need more than the number of bits you have, you don't have space for it. You can't just use more bits as HPACK will not know whether you are intending to use more bits (so should look at next byte) or if it's just a straight number (so only look at this one byte). It needs a signal to know that. That signal is using all 1s.

So to encode 40 in 5 bits, we need to use 11111 to say "it's not big enough", overflow to next byte. 11111 in binary is 31, so we know it's bigger than that, so we'll not waste that, and instead use it, and subtract it from the 40 to give 9 left to encode in the next byte. A new additional byte gives us 8 new bits to play with (well actually only 7 as we'll soon discover, as the first bit is used to signal a further overflow). This is enough so we can use 00001001 to encode our 9. So our complex number is represented in two bytes: XXX11111 and 00001001.

If we want to encode a value bigger than can fix in the first prefixed bit, AND the left over is bigger than 127 that would fit into the available 7 bits of the second byte, then we can't use this overflow mechanism using two bytes. Instead we use another "overflow, overflow" mechanism using three bytes:

For this "overflow, overflow" mechanism, we set the first byte bits to 1s as usual for an overflow (XXX11111) and then set the first bit of the second byte to 1. This leaves 7 bits available to encode the value, plus the next 8 bits in the third byte we're going to have to use (actually only 7 bits of the third byte, because again it uses the first bit to indicate another overflow).

There's various ways they could go have gone about this using the second and third bytes. What they decided to do was encode this as two numbers: the 128 mod, and the 128 multiplier.

   1337 = 31 + (128 * 10) + 26

So that means the frist byte is set to 31 as per pervious example, the second byte is set to 26 (which is 11010) plus the leading 1 to show we're using the overflow overflow method (so 100011010), and the third byte is set to 10 (or 00001010).

So 1337 is encoded in three bytes: XXX11111 100011010 00001010 (including setting X to whatever those values were).

Using 128 mod and multiplier is quite efficient and means this large number (and in fact any number up to 16,383) can be represented in three bytes which is, not uncoincidentally, also the max integer that can be represented in 7 + 7 = 14 bits). But it does take a bit of getting your head around!

If it's bigger than 16,383 then we need to do another round of overflow in a similar manner.

All this seems horrendously complex but is actually relatively simply, and efficiently, coded up. Computers can do this pretty easily and quickly.

It seems that i am left with a few useless variables, one being g

You are not print this value in the if statement. Only the left over value in the else. You need to print both.

which in the documentation is the 26 + 128 == 154. Lastly, where does 128 come from? I can't find any relation between the numbers besides the fact 2 raised to the 7th power is 128, but why is that significant? Is this because the first bit is reserved as a continuation flag? and an octet contains 8 bits so 8 - 1 = 7?

Exactly, it's because the first bit (value 128) needs to be set as per explanation above, to show we are continuing/overflowing into needing a third byte.