How to calculate sha256 faster in java?

Question

I have found out that calculating sha256 in java is slow. For example, it is slower than python. I wrote two simple benchmarks that calculate sha256 of 1GB of zeroes. In both cases the result is the same and correct, but the python time is 5653ms and the java time is 8623ms(53% slower). The result is similar every time and this is an important difference for me.

How to make the calculation in java faster?

Benchmarks:

Java:

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

public class BenchmarkSha256 {

  public static void main(String... args) throws NoSuchAlgorithmException {
    int size = 1024 * 1024;
    byte[] bytes = new byte[size];
    MessageDigest md = MessageDigest.getInstance("SHA-256");
    long startTime = System.nanoTime();
    for (int i = 0; i < 1024; i++)
      md.update(bytes, 0, size);
    long endTime = System.nanoTime();
    System.out.println(String.format("%1$064x", new java.math.BigInteger(1, md.digest())));
    System.out.println(String.format("%d ms", (endTime - startTime) / 1000000));
  }

}

Python:

#!/usr/bin/env python

import hashlib
import time

size = 1024 * 1024
bytes = bytearray(size)
md = hashlib.sha256()
startTime = time.time()
for i in range(0, 1024):
  md.update(bytes)
endTime = time.time()
print "%s\n%d ms" % (md.hexdigest(), (endTime - startTime) * 1000)

results:

~> java BenchmarkSha256
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
8623 ms

~> python BenchmarkSha256.py 
49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
5653 ms

versions of java and python:

~> java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

~> python --version
Python 2.7

Have you excluded the costs of starting up the environment in both cases, or are you interested in whole program times as opposed to algorithm times? (Both are valid things to ask about, but they're very different metrics.) — Donal Fellows, Oct 21 '11 at 08:31
possible duplicate of [Java: Calculate SHA-256 hash of large file efficiently](http://stackoverflow.com/questions/1741545/java-calculate-sha-256-hash-of-large-file-efficiently) — Donal Fellows, Oct 21 '11 at 08:36
@DonalFellows I am interested in the algorithm time only. The startup time is not so relevant at this scale. I tried to comment the "md.update(bytes, 0, size);" line in the java code and the time is 0.4s for the whole process except the calculation. Even with this time substracted, it is still slower. — Hristo Hristov, Oct 21 '11 at 08:36
@DonalFellows I think this is not a duplicate, because in the referenced question/answer there is a lot of file I/O involved, here I am interested in the pure hash calculation. — Hristo Hristov, Oct 21 '11 at 08:40
I suspect the digest uses Java rather than an assembly library. You could use http://www.cryptopp.com/benchmarks.html from Java. — Peter Lawrey, Oct 21 '11 at 08:45

score 5 · Answer 1 · edited Jun 20 '20 at 09:12

I ran a test on the following SHA-256 implementations: Java built-in, Groovy built-in, Apache Commons, Guava, and Bouncy Castle. My results on one run are here:

>groovy hash_comp.groovy
Hashing 1000000 iterations of SHA-256
time java: 2688         372023.8095238095 hashes/sec
time groovy: 1948       513347.0225872690 hashes/sec
time apache: 867        1153402.5374855825 hashes/sec
time guava: 953         1049317.9433368311 hashes/sec
time bouncy: 1890       529100.5291005291 hashes/sec

This was run on an Intel i5 8th Gen. Apache and Guava were easily to two fastest implementations. Apache Commons narrowly beat out Guava on 9/10 of my runs. My code for this test is available here.

Note that after running this test I started to wonder if you could go even faster by tapping into the CPU instruction set (Intel has SHA extensions). I'm not sure there is a JVM way to do this without JNI or JNA. I created another question here.

Update: Another option I found is the Amazon Corretto Crypto Provider (ACCP). Code available here.

What exactly is ACCP?

ACCP implements the standard Java Cryptography Architecture (JCA) interfaces and replaces the default Java cryptographic implementations with those provided by libcrypto from the OpenSSL project. ACCP allows you to take full advantage of assembly-level and CPU-level performance tuning, to gain significant cost reduction, latency reduction, and higher throughput across multiple services and products, as shown in the examples below.

score 5 · Answer 2 · answered Oct 21 '11 at 08:33

5

Have you tried feeding in the data incrementally? You can use messageDigest.update() with the bytes and then get the final digest with messageDigest.digest()?

Allocating a 1GB array in memory is a fairly chunky operation. You may find that smaller incremental updates are faster in the end.

answered Oct 21 '11 at 08:33

Jeff Foster

43,770
11
86
103

I have modified the java version to measure only the time for the calculation, the result is basically the same. I will try the incremental update variant and will post the results. – Hristo Hristov Oct 21 '11 at 08:54
Now the code is updated to do exactly this: incremental feeding and final digest, the issue is still the same: java is slower. – Hristo Hristov Oct 21 '11 at 10:24
We should probably give Java more of a chance and let it JIT optimize. Perhaps timing the code running a few hundred times times (in the same VM launch) will show some differences. Probably increasing the default heap size will help too (less GC pressure)? Is the Python implementation just a thin wrapper over a C library? – Jeff Foster Oct 21 '11 at 11:01
The code "md.update(...)" is already running 1024 times... I have tried increasing the heap size - no effect. I don't know about the python implementation, I just took the most straight-forward ways for doing it on both platforms. – Hristo Hristov Oct 21 '11 at 11:19
@JeffFoster Yes afaik python uses a thin wrapper around OpenSSL, so we're basically benchmarking the difference between OpenSSL's C implementation and Java (which presumably isn't implemented using JNI?) – Voo Oct 22 '11 at 10:12
@Voo we are searching for faster java solution, the benchmark is just to compare to something. – Hristo Hristov Oct 24 '11 at 06:04

score 2 · Answer 3 · answered Oct 21 '11 at 08:35

2

Well, unless you are doing this to compare two command line programs, this is not the best test. Primarily, these numbers are being polluted by the vast differences in overhead associated with each program. VM start times will vary. Memory allocation speeds will vary.

To clean this up a bit, simply take two time samples before and after each actual MD5 calculation within the code itself.

This will actually measure performance of the hashing operation itself.

answered Oct 21 '11 at 08:35

allingeek

1,378
8
16

Thanks, I did that, now the java version measures the time only for the hash calculation. The question is updated and still valid. – Hristo Hristov Oct 21 '11 at 08:53
@Hristo: You've got to do the same for the Python code too, or it's _still_ not fair (but not fair the other way). – Donal Fellows Oct 21 '11 at 09:43
@allingeek I should note that your "answer" is actually a comment. This should be upvoted comment, but not an answer. – Hristo Hristov Oct 21 '11 at 11:00
@Hristo: Now you have valid figures, the question I said this was a dupe of tells you _why_. :-) – Donal Fellows Oct 21 '11 at 12:13
@DonalFellows The figures are, as expected, basically the same. That question tells me nothing :) – Hristo Hristov Oct 21 '11 at 12:30

score -1 · Answer 4 · answered Oct 21 '11 at 09:00

-1

While you might be able to improve the performance of the Java tool a bit, the Python implementation will usually be faster because it is likely delegating to assembled libraries which run with significantly better performance.

If your project does not have any other significant dependencies on Java, I'd recommend going with the Python implementation.

answered Oct 21 '11 at 09:00

allingeek

1,378
8
16

3

The project is java only. The python implementation is just for demonstration. The question is, how to calculate sha256 with java faster? – Hristo Hristov Oct 21 '11 at 10:25

How to calculate sha256 faster in java?

4 Answers4

Linked