I wish to encode arbitrary binary data in an array of bytes. However the valid values for the data in each encoded byte is in the range of 1-127 inclusive. In other words, invalid values of encoded bytes are 0 and 128-255. With other words, I can't use the most significant bit (it has to be zero) and the value cannot be 0. So there 126 usable values to encode with in each byte. I know how to solve this using 6 bits per byte, simply storing data in bits 2-7, leaving bit 1 always 1 and bit 8 always 0. But I'd like to do better.
I have a feeling this is a "base of" question, as my previous solution with 6 bits seem similar to base64, but with a different table. So is this what I would like to look into called base126 encoding?
Why I want to use this base126 encoding (?) scheme is because I'd like to improve my Lua scripts in a game called Dual Universe, where data transmission is limited between emitters and receivers. For some reason the game can't handle a string with a null character in it, and strings (which are used for transmission) are utf8 encoded so I don't want to touch the most significant bit (or it would expect multibyte character data). I want to send binary data over this string with these limitations.
I don't know if I am barking up the tree but I have a feeling I should kind of think of the data I want to encode as a large number and divide and modulus by 126 and add 1, but I don't know how to work large arrays like that.
I'd be happy with a pointer into looking somewhere dealing with storing data in this way, if possible. I grok some oop/procedual languages so it doesn't have to be lua specific. Meanwhile I'll look into base85, because that sounds similar in ways to what I want to do based on my own guesswork. Maybe I get some insights there.
