Hashing file in Python 3?

Question

In Python 2, one could hash a string by just running:

someText = "a"
hashlib.sha256(someText).hexdigest()

But in Python 3, it needs to be encoded:

someText = "a".encode("ascii")
hashlib.sha256(someText).hexdigest()

But when I try this with a file:

f = open(fin, "r")
sha = hashlib.sha256()
while True:
    data = f.read(2 ** 20).encode("ascii")
    if not data:
        break
    sha.update(data)
f.close()

I get this on many files:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 8: invalid continuation byte

I assume this is because it's a binary file, which likely can't be converted to ASCII.

How can I encode the file without this problem?

Answer 1

On Unix systems, in Python 2 there was no distinction between binary- and text-mode files, so it didn't matter how you opened them.

But in Python 3 it matters on every platform. sha256() requires binary input, but you opened the file in text mode. That's why @BrenBam suggested you open the file in binary mode.

Since you opened the file in text mode, Python 3 believes it needs to decode the bits in the file to turn the bytes into Unicode strings. But you don't want decoding at all, right?

Then open the file in binary mode, and you'll read byte strings instead, which is what sha256() wants.

By the way, your:

someText = "a".encode("ascii")
hashlib.sha256(someText).hexdigest()

can be done more easily in a related way:

hashlib.sha256(b"a").hexdigest()

That is, pass it the binary data directly, instead of bothering with encoding a Unicode string (which the literal "a" is).

Answer 2

Try opening the file in binary mode with open(fin, "rb").

Answer 3

I have programmed a module which is able to hash big files with different algorithms.

pip3 install py_essentials

Use the module like this:

from py_essentials import hashing as hs
hash = hs.fileChecksum("path/to/the/file.txt", "sha256")

Take a look at the documentation.