45

I'm trying to calculate the SHA-1 value of a file.

I've fabricated this script:

def hashfile(filepath):
    sha1 = hashlib.sha1()
    f = open(filepath, 'rb')
    try:
        sha1.update(f.read())
    finally:
        f.close()
    return sha1.hexdigest()

For a specific file I get this hash value:
8c3e109ff260f7b11087974ef7bcdbdc69a0a3b9
But when i calculate the value with git hash_object, then I get this value: d339346ca154f6ed9e92205c3c5c38112e761eb7

How come they differ? Am I doing something wrong, or can I just ignore the difference?

Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985
Ikke
  • 99,403
  • 23
  • 97
  • 120
  • 2
    You can't really ignore the difference if you plan to use the hashes together. – Matthew Scharley Dec 08 '09 at 21:16
  • Forgot to mention, just used git as a reference, not going to use them together. – Ikke Dec 08 '09 at 21:24
  • 1
    If the file could be quite large, you can process it a block at a time so you don't need the whole thing in RAM at once: http://stackoverflow.com/questions/7829499/using-hashlib-to-compute-md5-digest-of-a-file-in-python3 – rakslice Aug 01 '13 at 18:24
  • Possible duplicate of [How does git compute file hashes?](http://stackoverflow.com/questions/7225313/how-does-git-compute-file-hashes) Minimal examples ;-) – Ciro Santilli OurBigBook.com May 17 '16 at 10:14

2 Answers2

51

git calculates hashes like this: 

sha1("blob " + filesize + "\0" + data)

Reference

Community
  • 1
  • 1
Brian R. Bondy
  • 339,232
  • 124
  • 596
  • 636
32

For reference, here's a more concise version:

def sha1OfFile(filepath):
    import hashlib
    with open(filepath, 'rb') as f:
        return hashlib.sha1(f.read()).hexdigest()

On second thought: although I've never seen it, I think there's potential for f.read() to return less than the full file, or for a many-gigabyte file, for f.read() to run out of memory. For everyone's edification, let's consider how to fix that: A first fix to that is:

def sha1OfFile(filepath):
    import hashlib
    sha = hashlib.sha1()
    with open(filepath, 'rb') as f:
        for line in f:
            sha.update(line)
        return sha.hexdigest()

However, there's no guarantee that '\n' appears in the file at all, so the fact that the for loop will give us blocks of the file that end in '\n' could give us the same problem we had originally. Sadly, I don't see any similarly Pythonic way to iterate over blocks of the file as large as possible, which, I think, means we are stuck with a while True: ... break loop and with a magic number for the block size:

def sha1OfFile(filepath):
    import hashlib
    sha = hashlib.sha1()
    with open(filepath, 'rb') as f:
        while True:
            block = f.read(2**10) # Magic number: one-megabyte blocks.
            if not block: break
            sha.update(block)
        return sha.hexdigest()

Of course, who's to say we can store one-megabyte strings. We probably can, but what if we are on a tiny embedded computer?

I wish I could think of a cleaner way that is guaranteed to not run out of memory on enormous files and that doesn't have magic numbers and that performs as well as the original simple Pythonic solution.

Ben
  • 9,184
  • 1
  • 43
  • 56
  • On second thought, this may have issues if f.read() can't return the whole file (e.g., in the case of multi-gigabyte files) and so it should probably iterate over chunks. – Ben Apr 17 '14 at 12:44