File comparison in VB.Net

Question

I need to know if two files are identical. At first I compared file sizes and creation timestamps, but that's not reliable enough. I have come up with the following code, that seems to work, but I'm hoping that someone has a better, easier or faster way of doing it.

Basically what I am doing, is streaming the file contents to byte arrays, and comparing thier MD5 hashes via System.Security.Cryptography.

Before that I do some simple checks though, since there is no reason to read through the files, if both file paths are identical, or one of the files does not exist.

Public Function CompareFiles(ByVal file1FullPath As String, ByVal file2FullPath As String) As Boolean

    If Not File.Exists(file1FullPath) Or Not File.Exists(file2FullPath) Then
        'One or both of the files does not exist.
        Return False
    End If

    If String.Compare(file1FullPath, file2FullPath, True) = 0 Then
        ' fileFullPath1 and fileFullPath2 points to the same file...
        Return True
    End If

    Dim MD5Crypto As New MD5CryptoServiceProvider()
    Dim textEncoding As New System.Text.ASCIIEncoding()

    Dim fileBytes1() As Byte, fileBytes2() As Byte
    Dim fileContents1, fileContents2 As String
    Dim streamReader As StreamReader = Nothing
    Dim fileStream As FileStream = Nothing
    Dim isIdentical As Boolean = False

    Try

        ' Read file 1 to byte array.
        fileStream = New FileStream(file1FullPath, FileMode.Open)
        streamReader = New StreamReader(fileStream)
        fileBytes1 = textEncoding.GetBytes(streamReader.ReadToEnd)
        fileContents1 = textEncoding.GetString(MD5Crypto.ComputeHash(fileBytes1))
        streamReader.Close()
        fileStream.Close()

        ' Read file 2 to byte array.
        fileStream = New FileStream(file2FullPath, FileMode.Open)
        streamReader = New StreamReader(fileStream)
        fileBytes2 = textEncoding.GetBytes(streamReader.ReadToEnd)
        fileContents2 = textEncoding.GetString(MD5Crypto.ComputeHash(fileBytes2))
        streamReader.Close()
        fileStream.Close()

        ' Compare byte array and return result.
        isIdentical = fileContents1 = fileContents2

    Catch ex As Exception

        isIdentical = False

    Finally

        If Not streamReader Is Nothing Then streamReader.Close()
        If Not fileStream Is Nothing Then fileStream.Close()
        fileBytes1 = Nothing
        fileBytes2 = Nothing

    End Try

    Return isIdentical
End Function

Answer 1

I would say hashing the file is the way to go, It's how I have done it in the past.

Use Using statements when working with Streams and such, as they clean themselves up. Here is an example.

Public Function CompareFiles(ByVal file1FullPath As String, ByVal file2FullPath As String) As Boolean

If Not File.Exists(file1FullPath) Or Not File.Exists(file2FullPath) Then
    'One or both of the files does not exist.
    Return False
End If

If file1FullPath = file2FullPath Then
    ' fileFullPath1 and fileFullPath2 points to the same file...
    Return True
End If

Try
    Dim file1Hash as String = hashFile(file1FullPath)
    Dim file2Hash as String = hashFile(file2FullPath)

    If file1Hash = file2Hash Then
        Return True
    Else
        Return False
    End If

Catch ex As Exception
    Return False
End Try
End Function

Private Function hashFile(ByVal filepath As String) As String
    Using reader As New System.IO.FileStream(filepath, IO.FileMode.Open, IO.FileAccess.Read)
        Using md5 As New System.Security.Cryptography.MD5CryptoServiceProvider
            Dim hash() As Byte = md5.ComputeHash(reader) 
            Return System.Text.Encoding.Unicode.GetString(hash) 
        End Using
    End Using
End Function

Answer 2

This is what md5 is made for. You're doing it the right way. However, if you really want to improve it further, I can recommend some things to explore. The emphasis is on explore, because none of these are slam dunks. They may help, but they may also hurt, or they may be overkill. You'll need to evaluate them for your situation and determine (through testing) what will be the best solution.

The first recommendation is to compute the md5 hash without loading the entire file into RAM. The example is C#, but the VB.Net translation is fairly straightforward. If you're working with small files, then what you already have may be fine. However, for anything large enough to end up on .Net's Large Object Heap (85,000 bytes), you probably want to consider using the stream technique instead.

Additionally, if you're using a recent version of .Net, you might want to explore doing this asynchronously for each file. As a practical matter, I suspect you'll get best performance from what you have, as the disk I/O is likely to be the slowest part of this, and I'd expect traditional disks to perform best if you allow them to read from the files in sequence, rather than making your disk seek back and forth between the files. However, you may still be able to do better with asynchronous methods, especially if you follow the previous suggestion, because you can also await at the Read() call level, in addition to awaiting for the entire file. Also, if you're running this on an SSD, that would minimize the problems with seeks and could make an asynchronous solution a clear winner. One warning, though: this is a deep rabbit hole to chase... one that can be worthwhile, but you can also end up spending a lot of time on a YAGNI situation. This is the kind of thing, though, you might choose to explore once for a situation where you probably won't use it, so that you understand it well enough to know how it can help in the future for those situations when you do need it.

One more point is that, for the asynch recommendation to work, you need to isolate the hashing code into it's own method... but you should probably do this anyway.

My final recommendation is to remove the File.Exists() checks. This is a tempting test, I know, but it's almost always wrong. Especially if you adopt the first recommendation, just open the streams near the top of the method using an option that fails if the file does not exist, and make your check on whether the stream opened or not.