Convert arbitrary int to bytes

Question

How can I use struct to convert an arbitrary int (unsigned, signed, long long,... just int in Python) to a byte sequence (bytes)?

If I read this correctly, I would need to decide on a format string depending on the sign or length of the int, but in Python I don't really have this distinction. I'm trying to convert an arbitrary int to bytes and re-create the int again from the sequence of bytes (bytes).

Here are some attempts which failed:

# int is too big
>>> struct.unpack('>i', struct.pack('>i', -12343243543543534))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
>>> -12343243543543534
-12343243543543534
>>> struct.unpack('>q', struct.pack('>q', -12343243543543534))
(-12343243543543534,)

# again, integer value is too big, but can be represented as integer (below)
>>> struct.unpack('>q', struct.pack('>q', -1234324354354353432432424))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: int too large to convert
>>> -1234324354354353432432424
-1234324354354353432432424

Alternatively, I could also use the largest "container" to convert to bytes and "up cast" when turning the bytes back into an integer, but then I would know which format string is safe (=largest) to use.

bytes approach

bytes(int) seems to have the same problem and requires to know about the sign:

>>> bytes(i)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: negative count
>>> i
-123432432432232

int.to_bytes / int.from_bytes

With a sufficiently large number of bytes, I can store "any" integer value, but it is still required to know about the sign.

>>> int(-1234324354354353432432424).to_bytes(64, 'little')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: can't convert negative int to unsigned

>>> int(-1234324354354353432432424).to_bytes(64, 'little', signed=True)
b'\xd9\xf0;\xd5l5\x86$\x9f\xfa\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'

>>> int.from_bytes(int(-1234324354354353432432424).to_bytes(64, 'little', signed=True))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: from_bytes() missing required argument 'byteorder' (pos 2)

# no sign parameter
>>> int.from_bytes(int(-1234324354354353432432424 + 1).to_bytes(64, 'little', signed=True), 'little')
13407807929942597099574024998205846127479365820592393377723561443721764030073546976801874298166903427690031858186486050853753882810712245592079295573651673

# actual number
>>> int.from_bytes(int(-1234324354354353432432424 + 1).to_bytes(64, 'little', signed=True), 'little', signed=True)
-1234324354354353432432423

Answer 1

bytes(int) seems to have the same problem and requires to know about the sign

No; this is not for the kind of conversion you have in mind at all. Observe:

>>> bytes(3) # 3 is the *length* of the result
b'\x00\x00\x00'

With a sufficiently large number of bytes, I can store "any" integer value, but it is still required to know about the sign.

This is not a limitation you could possibly avoid even in theory. The stored byte data is just data; whether it represents a signed or unsigned value is an interpretation of that data that must be imposed upon it. There is no way in principle that you could take in b'\x80' and just know whether it should represent 128 or -128, just by looking at it; and it does not matter whether that value came from using the struct module, int.to_bytes or anything else. No tool you use can make the decision for you, because you are not giving it any information with which the decision could be made.

(Note that your use of the struct module does encode assumptions about signedness. For example, q denotes a signed 64-bit type; for the corresponding unsigned type, you would use Q.)

In short, what you are asking for does not make sense.

You can, of course, work around the limitation by either a) adopting a convention (you know whether to interpret the value as signed or unsigned because of the context in which you are interpreting the bytes), or b) explicitly adding that information (storing a byte that encodes the signedness of the value - but this hardly ever is done in the real world). That is to say: you can give your decoding mechanism the information to make the decision, or you can make the decision yourself.

---

You can't just write this int into a file and read it from that file without also storing the sign (you need to specify a signed parameter in order to reassemble this integer correctly). I was hoping to "dump" the memory content of this integer into a file (as bytes) and read it again without worrying about size of sign.

You are, presumably, storing multiple values in the same file. That means that the foregoing is not a real limitation anyway - because you already have to make decisions about the size of the values you read from the file; where one ends and the next begins. Which is to say, in your example:

int.from_bytes(int(-1234324354354353432432424 + 1).to_bytes(64, 'little', signed=True), 'little', signed=True)

you tell yourself that you don't need to know the size for the .from_bytes call, but in a real application you still would - because you would be taking a slice of the file data, not the whole thing, and you would need to know the bounds of the slice.