I am trying to create a [single] md5 hash of multiple strings [in Java]. That is I want
md5(string1, string2, string3, ..., stringN)
Currently I am trying to concatenate all strings with some rarely used separator like #. That is
md5(string1#string2#...#stringN)
This looks hacky and I am worried about some weird string actually having the separator as part of it. What is the best way to do it?
This could possibly be better:
md5(md5(string1) + md5(string2) + ... + md5(stringN))
It'd eliminate the separator problem, but it's hard to say how good it is.
It does not really matter if the separator is part of the string. You probably dont even need a separator, since you're not going to be decomposing the concatenated string into parts
I've run into a similar problem before, and the best solution I could come up with was to use a non-typeable ascii character as the separator. Look at "man ascii" and pick one. My favorite is '\a', which is the ASCII symbol for the "bell" sound.
If you want to be sure that there can't be collision from just moving text from one string to the next, I recommend this scheme:
md5(<len1>+str1+<len2>+str2...)
Here, len1 is a fixed-length representation of the length of str1. For md5, it would be most appropriate to use a four-byte int value (assuming you know that you wont have strings longer than 2**31). Alternatively, use "decimal length#", i.e. (in Python notation)
md5(str(len(str1))+"#"+str(len(str2))+"#"+str2+...)
This can't produce collisions by just moving text from one string to the other, since the lengths would change.
for strings, I think +Coly Klein solution of adding non type-able characters is the best.
If you want a solution that works for binary data too, or you don't sure the string don't contain those characters, you can use recursive hash, e.g:
md5(md5(str1)+md5(str2)+md5(str3)+...+)
depends on the amount of data this solution can demand a lot of resources (not far ago i profiled a program and found that 97% of it's time it calculates sha1's, so I must warn you..)
Putting all the answers together, here is a class with a single public and static method which solves the posed question in an efficient manner. Feel free to comment, critique, or use this code as you wish (public domain and all)...
import java.nio.charset.Charset;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
/**
* MD5Summer is a utility class that abstracts the complexity of computing
* the MD5 sum of an array of Strings.
* <p>
* Submitted as an answer to the StackOverflow bounty question:
* <a href="http://stackoverflow.com/questions/4785275/compute-md5-hash-of-multi-part-data-multiple-strings">
* compute md5 hash of multi part data (multiple strings)</a>
* <p>
* This solution uses the 'fast' "byte[] to hex string" mechanism described here in
* <a href="http://stackoverflow.com/questions/9655181/convert-from-byte-array-to-hex-string-in-java">
* Convert from byte array to hex string in java</a>.
* <p>
* The MD5 sum is always calculated by converting the inputStrings to bytes based on
* the UTF-8 representation of those Strings. Different platforms using this class
* will thus always calculate the same MD5sum for the same Java Strings.
* <p>
* Using a ThreadLocal for storing the MessageDigest instance significantly reduces the amount of time spent
* obtaining a Digest instance from the java.security subsystem.
* <p>
* <i>Copyright - This code is released in to the public domain</i>
*/
public final class MD5Summer {
/**
* Calculate the MD5 sum on the input Strings.
* <p>
* The MD5 sum is calculated as if the input values were concatenated
* together. The sum is returned as a String value containing the
* hexadecimal representation of the MD5 sum.
* <p>
* The MD5 sum is always calculated by converting the inputStrings to bytes based on
* the UTF-8 representation of those Strings. Different platforms using this class
* will thus always calculate the same MD5sum for the same Java Strings.
*
* @param values The string values to calculate the MD5 sum on.
* @return the calculated MD5 sum as a String of hexadecimal.
* @throws IllegalStateException in the highly unlikely event that the MD5 digest is not installed.
* @throws NullPointerException if the input, or any of the input values is null.
*/
public static final String digest(final String ...values) {
return LOCAL_MD5.get().calculateMD5(values);
}
/**
* A Thread-Local instance of the MD5Digest saves construct time significantly,
* while avoiding the need for any synchronization.
*/
private static final ThreadLocal<MD5Summer> LOCAL_MD5 = new ThreadLocal<MD5Summer>() {
@Override
protected MD5Summer initialValue() {
return new MD5Summer();
}
};
private static final char[] HEXCHARS = "0123456789abcdef".toCharArray();
private static final Charset UTF8 = Charset.forName("UTF-8");
private final MessageDigest md5digest;
/**
* Private constructor - cannot create instances of this class from outside
*/
private MD5Summer () {
// private constructor making only thread-local instances possible.
try {
md5digest = MessageDigest.getInstance("MD5");
} catch (NoSuchAlgorithmException e) {
// MD5 should always be available.
throw new IllegalStateException("Unable to get MD5 MessageDigest instance.", e);
}
}
/**
* Private implementation on the Thread-local instance.
* @param values The string values to calculate the MD5 sum on.
* @return the calculated MD5 sum as a String of hexadecimal bytes.
*/
private String calculateMD5(final String ... values) {
try {
for (final String val : values) {
md5digest.update(val.getBytes(UTF8));
}
final byte[] digest = md5digest.digest();
final char[] chars = new char[digest.length * 2];
int c = 0;
for (final byte b : digest) {
chars[c++] = HEXCHARS[(b >>> 4) & 0x0f];
chars[c++] = HEXCHARS[(b ) & 0x0f];
}
return new String(chars);
} finally {
md5digest.reset();
}
}
}
I have compared the results of this program against Linux's md5sum program with the following little test:
public class MD5Tester {
// [rolf@rolfl ~/md5data]$ echo "Frodo Baggins" >> frodo
// [rolf@rolfl ~/md5data]$ echo "Bilbo Baggins" >> bilbo
// [rolf@rolfl ~/md5data]$ cat frodo bilbo
// Frodo Baggins
// Bilbo Baggins
// [rolf@rolfl ~/md5data]$ cat frodo bilbo | md5sum
// a8a25988435405b9a62634c887287b40 *-
// [rolf@rolfl ~/md5data]$
public static void main(String[] args) {
String[] data = {"Frodo Baggins\n", "Bilbo Baggins\n"};
String md5data = MD5Summer.digest(data);
System.out.println("Expect a8a25988435405b9a62634c887287b40");
System.out.println("Got " + md5data);
if (!"a8a25988435405b9a62634c887287b40".equals(md5data)) {
System.out.println("Data does not match!!!!");
}
}
}
you can use base64 to encode your String before you append your string. then in md5 function split your string and decode it. for example
public class MutilMd5 {
public static void main(String[] args) throws Base64DecodingException {
String s1 = "12#3";
String s2 = "#12345";
multMd5(Base64.encode(s1.getBytes()) + "#" + Base64.encode(s2.getBytes()));
}
public static void multMd5(String value) throws Base64DecodingException {
String md5 = "";
String[] encodeStrings = value.split("#");
if (encodeStrings != null) {
for (String encodeString : encodeStrings) {
System.out.println(new String(Base64.decode(encodeString.getBytes())));
md5 = md5 + DigestUtils.md5Hex(encodeString);
}
}
System.out.println(md5);
}
}
output is
12#3
13094636ff02b51be53c496d04d39bc2375704c2e00da07d2c9acc7646b2a844
Just don't separate them. It's a hash method: there is no use in separating them...
MessageDigest md5 = MessageDigest.getInstance("MD5");
byte[] bytes = ...;
for (String toHash: stringsToHash) {
md5.update(toHash.getBytes("UTF-8"));
}
md5.digest(bytes);
You can use MessageDigest class to generate your code. If I were you, if at a preprocess stage I know the length of each String, I would pass them as a unique String. If the Strings you're passing have different random lengths that can't be known I would hash them one by one, but I need to know if the origin entity and the receipt entity are well syncronized to know the length of the messages they are passing between each other.
private static final char[] HEXADECIMAL = { '0', '1', '2', '3',
'4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f' };
public String hash(String stringToHash) {
try {
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] bytes = md.digest(stringToHash.getBytes());
StringBuilder sb = new StringBuilder(2 * bytes.length);
for (int i = 0; i < bytes.length; i++) {
int low = (int)(bytes[i] & 0x0f);
int high = (int)((bytes[i] & 0xf0) >> 4);
sb.append(HEXADECIMAL[high]);
sb.append(HEXADECIMAL[low]);
}
return sb.toString();
} catch (NoSuchAlgorithmException e) {
//exception handling goes here
return null;
}
}
From my understanding, you want to hash lists of strings ensuring that no two different lists give the same result. This can be solved without thinking of the hash function at all.
You need a function String f(List<String> l) where no two input values result in the same output (an injective function from List<String> to String). With this, you can give the output to your hash function, and be ensured that there will be no collisions as far as the hash function itself ensures (note MD5 was broken years ago, so it may not be an appropriate choice). Here are 2 ways to implement f:
The most straightforward way is to just map every input to a subset of the character set of String that does not include your separator character:
public static String hex(String s) {
try {
String o = "";
for(byte b: s.getBytes("utf-8"))
o += String.format("%02x", b&0xff);
return o;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
public static String f(String... l) {
if (l.length == 0) return "";
String o = hex(l[0]);
if (l.length == 1) return o;
for (int i = 1; i < l.length; i++) o += "#" + hex(l[i]);
return o;
}
f("a#","b") => 6123#62
f("a","#b") => 61#2362
This is also pretty simple, but has the disadvantage that it can't be rewritten to work in a stream.
public static String prefix(String s) {
return s.length() + "." + s;
}
public static String f(String... l) {
if (l.length == 0) return "";
String o = prefix(l[0]);
if (l.length == 1) return o;
for (int i = 1; i < l.length; i++) o += "#" + prefix(l[i]);
return o;
}
f("a#","b") => 2.a##1.b
f("a","#b") => 1.a#2.#b
As You are using string as input and We know strings do not have NULL character so may use NULL character as separator for all Strings. You check input for NULL character while authentication.
md5(String1+NULL+String2+NULL+String3....)
Saves you time of multiple Md5.
