(This assumes you're doing it in Powershell, but these techniques apply to any language.)
I recommend checking file sizes first.
Do this first, it's quick!
if ((gci $file1).Length -ne (gci $file2).Length)
{
Write-Host "Files are different!"
}
else
{
# Same size, so compare contents...
}
Finally, you can do full-blown compare. If you're in PowerShell, take a look at Compare-Object
(alias diff
). For example,
if (diff (gc $file1) (gc $file2))
{
Write-Host "Files are different!"
}
It might be faster to do a buffered byte-to-byte comparison, as seen here:
http://keestalkstech.blogspot.com/2010/11/comparing-two-files-in-powershell.html
Alternatives:
An MD5 comparison might actually be slower than a byte-to-byte comparison. Not only do you need to read in the files, but then you also have to perform computation to get the hash. You can at least optimize by caching the hash of the old file--saving half the I/O.
Cause you're exporting a database table, most databases add rows to the end. You'll have to make sure this is your case, and that you're just adding and not updating. If so, you can just compare the last row in your file; e.g. the last 4K or however big your row size is.