Get SHA1 of HexString like HashCalc does

Question

HashCalc has, at the top, a field called "Data Format" I switch that to "HexString" then type in test value of 9a in the data field. I run a SHA-1 hash on it and the answer is:13cba177bcfad90e7b3de70616b2e54ba4bb107f

(NOTE: online hashers will hash "9a" as a string resulting in an answer of e8eef065fb7295044d65b305bab18a9a645d1abf. Which is wrong for this application)

Now, I need to embed this type of hashing into my Java program. This is what I got so far (wrapped in try/catch):

String ss = "9a";
ByteBuffer bb = ByteBuffer.allocate(8);  
byte[] ba = bb.putLong(Long.decode("0x"+ss).longValue()).array();
MessageDigest md = MessageDigest.getInstance("SHA-1");
String results = encodeHex(md.digest(ba));
System.out.println("sha:"+results);

However, my result is E73C417858807239DD5BC30BA978C14D57F80834

What am I doing wrong?

EDIT: Added Hex tag back, it seems obvious that the data has to be in some sort of hex format. As HashCalc has to be set to "HexString" and not "TextString" which returns a different result. And the solution could well include a change to how Im dealing with these hex numbers. --> which turned out to be true

Answer 1

DISCLAIMER

The OP (Pimp Trizkit), is the one who found the right solution. I just used his solution (with minor change) to showcase the result and to have fun. All rights reserved)

Also, the bytes array to hex string converting algorithm offered by the OP is much faster, than the ones in my example code. See his solution for the implementation.
(For the details read the comments down below)

---

One of the manual solutions:
(Important! It was my first answer, but it's only for getting hash from the text string, not from the hex string as the OP asked. See the update below):

import java.security.MessageDigest;

public class TestHash {

    public static void main(String[] args) throws Exception {
        String password = "9a";

        MessageDigest md = MessageDigest.getInstance("SHA-1");
        md.update(password.getBytes());
        byte[] byteData = md.digest();
        // byte[] byteData = md.digest(password.getBytes());    // both updates and completes the hash computation

        // Method 1 of converting bytes to hex format
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < byteData.length; i++) {
            sb.append(Integer.toString((byteData[i] & 0xFF) + 0x100, 16).substring(1));
        }

        System.out.println("1) Hex format : " + sb.toString());

        // Method 2 of converting bytes to hex format
        StringBuffer hexString = new StringBuffer();
        for (int i = 0; i < byteData.length; i++) {
            String hex = Integer.toHexString(0xff & byteData[i]);
            // NB! E.g.: Integer.toHexString(0x0C) will return "C", not "0C"            
            if (hex.length() == 1) {
                hexString.append('0');
            }
            hexString.append(hex);
        }
        System.out.println("2) Hex format : " + hexString.toString());      
    }
}

Output:

1) Hex format : e8eef065fb7295044d65b305bab18a9a645d1abf
2) Hex format : e8eef065fb7295044d65b305bab18a9a645d1abf

---

UPDATE

Based on the OP found solution, here is the code, that showcases getting the SHA-1 hash from a hex string instead of a text string. Plus it presents several methods of manually converting byte arrays to hex string (just for fun). Sorry, I was in a mood ))

See my comments inside the main method and inside different bytesToHexString() methods for the explanation of important moments.

import java.security.MessageDigest;
import java.math.BigInteger;
import javax.xml.bind.DatatypeConverter;

public class TestHash3 {

    public static void main(String[] args) throws Exception {
        String hexString = "1234";

        /*
         * NB!
         * Before passing hex string to DatatypeConverter.parseHexBinary(),
         * we need to check if the hex sting is even-length, 
         * otherwise DatatypeConverter.parseHexBinary() will throw a
         * java.lang.IllegalArgumentException: hexBinary needs to be even-length
         */
        hexString = (hexString.length() % 2 == 0) ? hexString : "0" + hexString;
        byte[] bytes = DatatypeConverter.parseHexBinary(hexString);

        MessageDigest md = MessageDigest.getInstance("SHA-1");
        byte[] byteData = md.digest(bytes);

        System.out.println("1) SHA-1 hash for the hex string " + hexString + ": " +
                            bytesToHexString1(byteData));
        System.out.println("2) SHA-1 hash for the hex string " + hexString + ": " +
                            bytesToHexString2(byteData));
        System.out.println("3) SHA-1 hash for the hex string " + hexString + ": " +
                            bytesToHexString3(byteData));
        System.out.println("4) SHA-1 hash for the hex string " + hexString + ": " +
                            bytesToHexString4(byteData));
    }

    public static String bytesToHexString1(byte[] bytes) {
        StringBuffer hexBuffer = new StringBuffer();
        for (int i = 0; i < bytes.length; i++) {
            hexBuffer.append(Integer.toString((bytes[i] & 0xFF) + 0x100, 16).substring(1));
        }

        return hexBuffer.toString();
    }

    public static String bytesToHexString2(byte[] bytes) {
        StringBuffer hexBuffer = new StringBuffer(bytes.length * 2);
        for (byte b: bytes) {
            int n = b & 0xFF;   // casting to integer to avoid problems with negative bytes
            if (n < 0x10) {
                hexBuffer.append("0");
            }
            hexBuffer.append(Integer.toHexString(n));
        }

        return hexBuffer.toString();
    }       

    public static String bytesToHexString3(byte[] bytes) {
        StringBuffer hexBuffer = new StringBuffer();
        for (int i = 0; i < bytes.length; i++) {
            String hexString = Integer.toHexString(0xff & bytes[i]);
            // NB! E.g.: Integer.toHexString(0x0C) will return "C", not "0C"            
            if (hexString.length() == 1) {
                hexBuffer.append('0');
            }
            hexBuffer.append(hexString);
        }

        return hexBuffer.toString();
    }

    public static String bytesToHexString4(byte[] bytes) {
        String hexString = new BigInteger(1, bytes).toString(16);

        /*
         * NB!
         * We need an even-length hex string to propely represent bytes in hexadecimal.
         * A hexadecimal representation of one byte consists of two hex digits.
         * If the value is less than 16 (dec), it is prepended with zero
         * E.g.:
         * 1  (byte)    ==> 01 (hex)    // pay attention to the prepended zero
         * 15 (byte)    ==> 0F (hex)
         * 16 (byte)    ==> 10 (hex)    // no need to prepend
         * 255(byte)    ==> FF (hex)
         *
         * BigInteger.toString(16) can return both even and odd-length hex strings.
         * E.g.:
         * byte[] bytes = {15, 16}  // two bytes
         * BigInteger(1, bytes).toString(16) will produce (NB!): f10
         * But we need (NB!): 0f10
         * So we must check if the resulting hex string is even-length,
         * and if not, prepend it with zero.
         */
        return ((hexString.length() % 2 == 0) ? hexString : "0" + hexString);
    }
}

Output:

1) SHA-1 hash for the hex string 1234: ffa76d854a2969e7b9d83868d455512fce0fd74d
2) SHA-1 hash for the hex string 1234: ffa76d854a2969e7b9d83868d455512fce0fd74d
3) SHA-1 hash for the hex string 1234: ffa76d854a2969e7b9d83868d455512fce0fd74d
4) SHA-1 hash for the hex string 1234: ffa76d854a2969e7b9d83868d455512fce0fd74d

btw Checking if the hex string is even-length inside byteToHexString4() so it could be used independently.

---

UPDATE 2

The user @kan brought one more method of converting byte array to hex string, very simple one liner and the second fastest after the OP's method.

DatatypeConverter.printHexBinary(byte[] val)

Answer 2

I use a small utility class:

public abstract class Sha1Util
{
    private static final Charset UTF8 = Charset.forName("UTF-8");

    public static MessageDigest newSha1Digest()
    {
        try
        {
            return MessageDigest.getInstance("SHA-1");
        } catch (NoSuchAlgorithmException e)
        {
            throw new Error(e);
        }
    }

    public static void update(final MessageDigest digest, final String s)
    {
        digest.update(s.getBytes(UTF8));
    }

    public static String sha1sum(final MessageDigest digest)
    {
        return String.format("%040x", new BigInteger(1, digest.digest()));
    }
}

Test:

@Test
public void testSha1For9a()
{
    final MessageDigest md = SecUtil.newSha1Digest();
    SecUtil.update(md, "9a");// you could use several updates e.g. for salted passwords
    assertEquals("e8eef065fb7295044d65b305bab18a9a645d1abf", SecUtil.sha1sum(md));

}

Answer 3

I figured it out! Thanks to Louis Wasserman for inspiration in his comment to my question. When I used 000000000000009a with HashCalc it returned the same result as my function! I realized that I was forcing a certain data length (the length of a long). Turns out it needs to be an arbitrary length (like HashCalc... as I said, I needed to duplicate the HashCalc behavior, not just for 9a or what ever small test case we use, but for all possible data). Even longer than long. So these other solutions don't work here.

I got around this by converting the original hex string to a char[] array then concat them together in pairs and converting to a byte, in a loop, storing each in the byte[] that is passed to digest. This worked! Then I saw Kan's comment to his answer on this, and showed me I was just reinventing the wheel there.

So here is the final code, and its amazingly the same size as the original:

private String HexToSHA1(String ss) throws IllegalArgumentException {
    MessageDigest md = null;
    try { md = MessageDigest.getInstance("SHA-1"); }
    catch ( Exception e ) {}
    return byteArrayToHexString(md.digest(DatatypeConverter.parseHexBinary(ss)));
}

This runs pretty quick, not sure if there is even a faster way. Of course thats also based on how fast your byteArrayToHexString function is. Here is mine:

private String byteArrayToHexString(byte[] data) {
    char[] toDigits =  {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
    int l = data.length;
    char[] out = new char[l << 1];
    for (int i = 0, j = 0; i < l; i++) {
        out[j++] = toDigits[(0xF0 & data[i]) >>> 4];
        out[j++] = toDigits[0x0F & data[i]];
    }
    return new String(out);
}

And its pretty fast. Credit to someone somewhere. I didn't write it.

EDIT: Moving char[] toDigits = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}; to the global (or "up one in scope") makes it faster still.