How can I effectively store binary data in a file that's in a text format like CSV?

Question

I'm currently working on a password storage program in Python, though C would likely be faster. I've been trying for the past hour or so to find a way to store a bytes object in a CSV file. I'm hashing the passwords with their own salt, and then storing that, and grabbing it again to check the password. It works perfectly well when it's stored in memory.

salt = os.urandom(64)
hash = hashlib.pbkdf2_hmac(
    'sha256',
    password.encode('utf-8'),
    salt,
    1000000
)
storage = salt + hash
salt_from_store = storage[:64]
hash_from_store = storage[64:]

However, when I try storing it in a CSV file, so it doesn't have to be constantly running, I get an error,

TypeError: write() argument must be str, not bytes

So, I converted it to a string using,

str(storage)

and that wrote just fine. But then, when I get it from the file, it's still a string, and the length goes from 128 (bytes) to 300+ (chars). It's also never consistent. I don't know the encoding, so I can't change it like that, when I print the bytes, it's a bunch of characters with backslashes and X's

b'\xfd\x3a'

and occasionally some random special characters. I'm not sure if there's a way to convert that to an int, and let it be converted back. Another issue is that I've found a way to do it, by changing

b"\xf1\x96"

to

"b\xf1\x96"

which prints the encoded text, rather than the bytes it's made up of. However, I don't know if that's a good way of changing it, and if it is, if there's a way to do it without something like

bytes[0] = '"'
bytes[1] = 'b'

Answer 1

To write bytes, either write to something that expects to contain bytes, or write text that represents the bytes in some way. CSV is fundamentally a text-based format. If you're going to use a CSV file, then you're going to open it in text mode, and write text to it.

Fundamentally, every file on the hard drive consists of bytes. This implies that, when you open the CSV file, you will be choosing (or using a default) text encoding scheme. So your bytes object will have to be converted twice (to text, and then into the underlying bytes in the file - which you could verify for example with a hex editor) on writing, and twice again on reading. That's just the reality of dealing with mixed data. Thankfully, half that work is taken care of for you automatically (by the open call, or wrappers for that like csv.Reader).

So, I converted it to a string using str(storage)

This is not actually a conversion in the sense that you're most likely interested in. This is asking for a printable, human-readable representation of the object (There is also repr, which asks for a more technically-oriented representation. For str and bytes objects, that's where the enclosing quotation marks come from, among other adjustments. When you print something, its str is used. When you evaluate something at the REPL, you see the repr of the result - except that when the result is None, it doesn't show anything at all). Specifically for dealing with bytes and str objects, Python has a concept of encoding and decoding, which uses explicit .encode (str->bytes) and .decode (bytes->str) methods. These are topics you can easily look up in the documentation (or previous Stack Overflow questions, or on the Internet in general).

when I print the bytes, it's a bunch of characters with backslashes and X's

Yes, this is the form that Python uses to tell you what data exists inside the bytes object. What you're saying here is basically the same as "when I print the list, it's a bunch of list elements with commas surrounded by square brackets", or "when I print the integer, it's a bunch of digit symbols".

But then, when I get it from the file, it's still a string, and the length goes from 128 (bytes) to 300+ (chars).

So decode it again. Of course you do need to encode properly. Everything that you get from the file will be a string, because you are opening the file in text mode, because CSV is a text format. (Incidentally, you are using the csv standard library module for this, right?)

It's also never consistent. I don't know the encoding

So tell it which encoding to use; and if you need to use a consistent amount of text, choose an encoding that consistently maps one byte to one Unicode code point (such as latin-1, also named iso-8859-1). But I suspect you don't actually care how long the text is (if anything, you'd care about the amount of bytes used in the file).

I've found a way to do it, by changing

You can only do this with literal data. Do not think in these terms. The b is part of the language syntax. It is not data.

Answer 2

If you want to save bytes as a string, you should probably encode them in a format made for this like base64. This is more efficient with space than directly writing hex.

Trying to convert arbitrary bytes to an encoding like utf-8 directly will likely result in UnicodeDecodeError errors.

In your case, you could do something like:

import os, hashlib, base64

password = "top_secret"

salt = os.urandom(64)
hash = hashlib.pbkdf2_hmac(
    'sha256',
    password.encode('utf-8'),
    salt,
    1000000
)
storage = salt + hash

# convert to a base64 string:
s = base64.b64encode(storage).decode('utf-8')

print(s) # <-- string you can save this to a file

# after reading it back from a file convert back to bytes
the_bytes = base64.b64decode(s)

the_bytes == storage 
# True

Answer 3

You could use hex. Let's get some data:

>>> import os
>>> b = os.urandom(10)
>>> b
b'\xc5\xe2{\xdf\xd2\x13\xa7\x0b\xef\x07'

As a hex string that you can write to CSV:

>>> b.hex()
'c5e27bdfd213a70bef07'

Back to bytes:

>>> bytes.fromhex(b.hex())
b'\xc5\xe2{\xdf\xd2\x13\xa7\x0b\xef\x07'