Signed message using OpenSSL; can't verify with Android Java

Question

I'm using SHA256 and RSA to sign a message on my Ubuntu machine using OpenSSL. My goal is to verify this message on Android using Android's Java.

Following commands were used on ubuntu:

openssl genrsa -out private.pem 1024 
openssl rsa -in private.pem -out public.pem -outform PEM -pubout
echo 'foobar' > data.txt
openssl dgst -sha256 < data.txt > hash
openssl rsautl -sign -inkey private.pem -keyform PEM -in hash  > signature
openssl rsa -in private_key.pem -pubout -outform DER -out public_key.der
openssl enc -base64 -in signature -out base64_signature

I have now created keys, signed the message, created a .der file for the public key that should be able to be accessed in Java and encoded the message with Base64. I then place the .der public key on my device and successfully load the key into the class PublicKey.

This method is used to verify the message:

public static boolean verify(PublicKey publicKey,String data,String verification){
    java.security.Signature sig;
    try {
        sig = java.security.Signature.getInstance("SHA256WithRSA");
        sig.initVerify(publicKey);
        try {
            sig.update(verification.getBytes());
        } catch (Exception e) {
            ...
        }

        if (!sig.verify(Base64.decode(data, Base64.DEFAULT))) {
            return false;
        }
        return true;
    }
    catch ....
    return false;
}

Parameters when calling the method:

verify(PublicKey, Base64 encoded data in a String that is to be verified, "foobar");

Obviously the verification fails, but I can't understand why. I'm guessing it has to do something with the encoding(?).

Update! So I managed to write the results of Base64.decode(data, Base64.DEFAULT))to a file and compare it with the original signature file using a hexeditor. Completely different!

Text encoding may be a problem. You can check that by creating a SAH256 MessageDigest of your used `verification` and compare the output with the OpenSSL hash. You can also check the "content" of your signature by decrypting it in Java with RSA/ECB/NoPadding. — Robert, Jun 23 '15 at 14:17

Roman · Accepted Answer · 2015-06-25T12:21:38.757

Java produces and expects to receive signatures in slightly different form. Hash of the message must be encoded in DER, then padded with PKCS#1 and only then signed with private key. And Openssl has a command for that (because it's actually a standard procedure). Instead of

openssl dgst -sha256 < data.txt > hash
openssl rsautl -sign -inkey private.pem -keyform PEM -in hash  > signature

you do

openssl dgst -sha256 -binary -sign private.pem data.txt > signature

Also note:

your data.txt contains a newline, don't forget it in String verification variable
sig.update(verification.getBytes()) should explicitly indicate a charset - the same charset, that was used to fill the data.txt file, for example: sig.update(verification.getBytes("UTF-8"))

The rest of your commands/code seems OK.

UPD - to answer @GilCol about the differences:

The padding is the same for both signed messages (PKCS#1). But the messages are different.

When you use openssl dgst -sha256 < data.txt > hash, hash will contain (depending on openssl version):

(stdin)= aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f

or

aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f

It is just plain text and it is the message you will sign using openssl rsautl -sign .... We can see that with openssl rsautl -verify ...:

# raw message as-is - we can see the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff 00 61   ...............a #
00c0 - 65 63 30 37 30 36 34 35-66 65 35 33 65 65 33 62   ec070645fe53ee3b #
00d0 - 33 37 36 33 30 35 39 33-37 36 31 33 34 66 30 35   3763059376134f05 # your plain-text message
00e0 - 38 63 63 33 33 37 32 34-37 63 39 37 38 61 64 64   8cc337247c978add #
00f0 - 31 37 38 62 36 63 63 64-66 62 30 30 31 39 66 0a   178b6ccdfb0019f. # we can even see newline char (0a) at the end

# strip the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -hexdump
0000 - 61 65 63 30 37 30 36 34-35 66 65 35 33 65 65 33   aec070645fe53ee3
0010 - 62 33 37 36 33 30 35 39-33 37 36 31 33 34 66 30   b3763059376134f0
0020 - 35 38 63 63 33 33 37 32-34 37 63 39 37 38 61 64   58cc337247c978ad
0030 - 64 31 37 38 62 36 63 63-64 66 62 30 30 31 39 66   d178b6ccdfb0019f
0040 - 0a                                                .

If you use openssl dgst -sha256 -binary < data.txt > hash to get hash in binary (pure) form, and then sign it, the result will be better, but still not right:

# raw message as-is - we can see the same padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00d0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff 00   ................
00e0 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # the hash - now in binary form
00f0 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... #

# strip the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -hexdump
0000 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # just the hash, nothing else
0010 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... #

But when you use openssl dgst -sha256 -sign ..., the message is different - it's now a standard ASN.1 structure for message digests (hashes). Let's see:

# raw message as-is - we can see the same padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 31 30   .............010 #
00d0 - 0d 06 09 60 86 48 01 65-03 04 02 01 05 00 04 20   ...`.H.e.......  # the message - it's different
00e0 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # <- we can see the hash (in binary form) starting at this line
00f0 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... #

# strip the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -hexdump
0000 - 30 31 30 0d 06 09 60 86-48 01 65 03 04 02 01 05   010...`.H.e.....
0010 - 00 04 20 ae c0 70 64 5f-e5 3e e3 b3 76 30 59 37   .. ..pd_.>..v0Y7
0020 - 61 34 f0 58 cc 33 72 47-c9 78 ad d1 78 b6 cc df   a4.X.3rG.x..x...
0030 - b0 01 9f                                          ...

# parse the message and show the underlying ASN.1 structure
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -asn1parse 
    0:d=0  hl=2 l=  49 cons: SEQUENCE          
    2:d=1  hl=2 l=  13 cons:  SEQUENCE          
    4:d=2  hl=2 l=   9 prim:   OBJECT            :sha256               # type of hash
   15:d=2  hl=2 l=   0 prim:   NULL              
   17:d=1  hl=2 l=  32 prim:  OCTET STRING      
      0000 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # the hash in binary form
      0010 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... # and no extra newline chars

As you can see, only the last signature file had proper ASN.1 structure, previous two were just "some arbitrary" messages, signed with RSA private key.

Thank you, it solved my problem. I also found this thread: http://stackoverflow.com/questions/13419201/why-are-the-rsa-sha256-signatures-i-generate-with-openssl-and-java-different But I still don't understand the cryptographic difference between those commands. Is the padding different or what? — GilCol, Jun 25 '15 at 06:34

Signed message using OpenSSL; can't verify with Android Java

1 Answers1

Linked