Why does ComputeHash perform much slower then certutil -hashfile?

Question

I am looking for efficient way of calculating hash of big files (3GB) and realized that calling Windows certutil with argument -hashfile perform hash computation 4 time faster (16 sec) than doing it via SHA512.Create().ComputeHash (~60 sec) and I am not really understand such a big differences.

I was trying to 'play' with reading buffer size of FileStream but it gives me about 2 sec so not really important optimization.

1) Doing hash computation by ComputeHash:

var sw = Stopwatch.StartNew();
using (var fs = new FileStream(@"C:\Temp\BigFile.dat", FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize: 1024 * 1024 * 10, options: FileOptions.SequentialScan))
{
    Console.WriteLine(BitConverter.ToString(SHA512.Create().ComputeHash(fs)));
}
Console.WriteLine(sw.ElapsedMilliseconds);

2) Doing it via certutil

var sw = Stopwatch.StartNew();
var fPath = @"C:\Temp\BigFile.dat";
var p = Process.Start(new ProcessStartInfo(@"certutil", $"-hashfile \"{fPath}\" SHA512") { RedirectStandardOutput = true, UseShellExecute=false});
p.WaitForExit();
Console.WriteLine(p.StandardOutput.ReadToEnd());
Console.WriteLine(sw.ElapsedMilliseconds);

I would expect some difference in time due to managed/native nature of code but 16 sec vs 60 looks a bit confusing.

At this question it is also mentioned that native code works much faster but the are no explanation of this difference and I am mostly interested in understanding the difference. I was thinking that in nowadays it shouldnt be such a big difference in such simple example (IO + math?)

Possible duplicate of [What is the fastest way to create a checksum for large files in C#](https://stackoverflow.com/questions/1177607/what-is-the-fastest-way-to-create-a-checksum-for-large-files-in-c-sharp) — Pavel Anikhouski, Aug 01 '19 at 09:08
I don't think this is the duplicate of the question mentioned above. — Dialecticus, Aug 01 '19 at 09:09
Try one of the specific `SHA512` implementations? `SHA512Managed`, `SHA512Cng`, `SHA512CryptoServiceProvider`? — canton7, Aug 01 '19 at 09:09

Dialecticus · Accepted Answer · 2019-08-01T09:25:09.997

2

There are three implementations of SHA512 in .NET (and the default depends on CryptoConfig.AllowOnlyFipsAlgorithms property):

"System.Security.Cryptography.SHA512CryptoServiceProvider"
"System.Security.Cryptography.SHA512Cng"
"System.Security.Cryptography.SHA512Managed"

It could be that those implementations have different performances. Call SHA512.Create(string) with each of those strings, and compare results.

edited Aug 01 '19 at 09:25

answered Aug 01 '19 at 09:13

Dialecticus

16,400
7
43
103

(Or just do e.g. `new SHA512Cng()`) – canton7 Aug 01 '19 at 09:14
Just check `SHA512Managed` as well, in case `SHA512.Create()` was giving you `SHA512Cng`. Not that I've got particularly high hopes for it, but it's worth making sure. – canton7 Aug 01 '19 at 09:23
There is no `SHA512CryptoServiceProvider.Create` - you're just calling `SHA512.Create()`, which will give you `SHA512Managed` / `SHA512Cng` – canton7 Aug 01 '19 at 09:33
@PavelAsedovski try exactly as written `SHA512.Create(string)`, where `string` is one of the stated above, exactly as written. – Dialecticus Aug 01 '19 at 09:34
@canton7 I was busy writing and investigating. And failing at it as well. But, here's the comment upvote. – Dialecticus Aug 01 '19 at 09:36
SHA512.Create("System.Security.Cryptography.SHA512Managed") works as 56 sec but all other givs me about 16 sec. So I can say it is the answer - because this particular implementation is slow and two other - much faster. Thank you! – Pavel Asedovski Aug 01 '19 at 09:37
`SHA512Managed` is, as the name suggests, implemented in C#. `SHA512CryptoServiceProvider` invokes advapi32.dll, while `SHA512Cng` invokes bcrypt.dll. – canton7 Aug 01 '19 at 09:41

Why does ComputeHash perform much slower then certutil -hashfile?

1 Answers1