MD5 Hashes failing for large Strings

Question

I'm making a MD5 hash of a response, and then signing it with a shared secret.

For most calls this works, but strangely fails (Generates a MD5 hash different from the client) on the only two calls that bring a lot of content in the body.

Can this be because of the size of the body? or maybe because those calls return the content chunked?

Any idea will be appreciated. Thanks a lot.

The hashing code (note that algorithm == MD5 and ENCODING == 'UTF-8'):

private static byte[] hash(String toHash, String algorithm){
      try{
      MessageDigest dg = MessageDigest.getInstance(algorithm);
      dg.update(toHash.getBytes(ENCODING));
      return dg.digest();
    }catch(Exception e){
      throw new ApiInternalException("Error while hashing string: " + toHash,e);
    }
    }

can you post the code you use to generate the MD5, there are more ways than one. — harschware, Feb 09 '10 at 21:36
I recall your question: http://stackoverflow.com/questions/2181215/md5-signing-a-httpservletresponse How about if you actually try my answer? :) — BalusC, Feb 09 '10 at 21:43
@BalusC thanks but that's a totally different question. One is about how to read contents from a OutputStream and this is about MD5 signatures failing. Thanks anyway — Pablo Fernandez, Feb 09 '10 at 22:09
You should indeed be more specific about how it fails. What happens? Any exception? Which one? Trace please. — BalusC, Feb 09 '10 at 22:20
By fail I mean that the hash calculated by the client does not match the one generated by the server. And _no_ the server does not modify the body after generating the hash. — Pablo Fernandez, Feb 09 '10 at 22:34
Well, it really sounds like not **everything** has been written through *your* writer, as I initially already suspected (in my deleted answer). But OK, hope you get it nailed down. — BalusC, Feb 09 '10 at 23:07

score 2 · Answer 1 · edited Feb 09 '10 at 21:55

It'd be great if you included your code. Without that, I can only guess what the problem is. Anyway, here's the correct way to create an MD5 hash in Java. If your code differs from this, then you have a problem.

String plainString = "Hash me please";
String md5Hash = "NOTHASHED";
try {
     MessageDigest md5Digest = MessageDigest.getInstance("MD5");
     md5String = new String(md5Digest.digest(plainString.getBytes()));
} catch (NoSuchAlgorithmException nsae) {
     // MD5 is included in all versions of Java, this can never happen
}

Of course, this will return something that looks like: �ǚ��;�f��&fu If you want it to be human readable, it's usually suggested that you Bas64 encode it, in which case just use the line:

new String(Base64Encoder.encode((md5Digest.digest(DESKTOP_STRING.getBytes()))));

Which will give you something that looks like: ssea19zwO6Jm3AiF4SZmdQ==

Keeping in mind that you will need to unencode it later before using it as an md5 hash.

Erm, the nomal practice is that you format the bytearray as a hexstring. Also see my answer on the OP's previous question on this subject: http://stackoverflow.com/questions/2181215/md5-signing-a-httpservletresponse — BalusC, Feb 09 '10 at 21:44
I'm doing something similar but the encoding is in hexa instead of `Base64`, check the edit! Thanks a lot — Pablo Fernandez, Feb 09 '10 at 22:10

Stephen C · Answer 2 · 2010-02-09T22:31:56.293

Either explanation is plausible. Other possible explanation include:

something (maybe a proxy server) is altering content in transit,
there is a mismatch in the way that character encoding / decoding is being dealt with
it is actually the MD5 hashes that are being damaged.

You need to gather more information to figure out which it is. I suggest that you modify your client and server sides to capture the data being sent / received into files, move them to the same machine and do a byte-wise comparison. Other things you could try include turning off chunking and dumping / comparing the MD5 checksums at both ends.

EDIT : It would also help us help you if you posted the code that does the MD5 checksum calculation at both ends, and the code that encodes / decodes the checksums for transmission (e.g. using hexadecimal, base64 or whatever).

So it has nothing to do with the html response being `chunked`? — Pablo Fernandez, Feb 09 '10 at 22:06
@Pablo - as I said, either of your two proposed explanations is plausible. But you need to figure out which of the plausible explanations is the actual one. — Stephen C, Feb 09 '10 at 22:23

MD5 Hashes failing for large Strings

2 Answers2