OpenSSL - Proper RSA Signature Generation and Verification

Question

I'm learning some OpenSSL RSA usage. I noticed that there are two different ways of generating and verifying file signatures. One by using openssl-dgst(1) and the other using openssl-pkeyutl(1) and they both seem to verify, accept private and public certificates, output signature files, accept algorithms, but they are not interchangeable. A signature generated by openssl-pkeyutl will not work in verification as a signature with openssl-dgst, and the opposite seems to be true as well. Even though both signature files have similar data structure formats.

Note: this is regarding RSA certificates specifically. I am not completely sure if the same possibility exists with X.509 certificates.

Question: Which is the more proper usage? openssl-pkeyutl or openssl-dgst?

Method 1: openssl dgst

# directory/
#     test.txt      - File to create a signature for
#     cert.pem      - Private/Public RSA Key, encrypted with hmacWithSHA256 via PKCS#8
#     cert.pub.pem  - Public Key of cert.pem extracted with `openssl rsa -pubout...`
#     test.sig      - File signature created by OpenSSL

# Create test.sig
$ openssl dgst -sha256 -sign cert.pem -out test.sig test.txt
Enter pass phrase for cert.pem: test

# Verify with the private key
$ openssl dgst -sha256 -verify cert.pub.pem -signature test.sig test.txt
Verified OK

# Verify with the public key
$ openssl dgst -sha256 -prverify cert.pem -signature test.sig test.txt
Enter pass phrase for cert.pem: test
Verified OK

Method 2: openssl pkeyutl

# directory/
#     test.txt      - File to create a signature for
#     cert.pem      - Private/Public RSA Key, encrypted with hmacWithSHA256 via PKCS#8
#     cert.pub.pem  - Public Key of cert.pem extracted with `openssl rsa -pubout...`
#     test.sig      - File signature created by OpenSSL

# Create test.sig
$ openssl pkeyutl -sign -in test.txt -out test.sig -inkey cert.pem
Enter pass phrase for cert.pem: test

# Verify with the private key
$ openssl pkeyutl -verify -sigfile test.sig -in test.txt -inkey cert.pub.pem -pubin
Signature Verified Successfully

# Verify with the public key
$ openssl pkeyutl -verify -sigfile test.sig -in test.txt -inkey cert.pem
Enter pass phrase for cert.pem: test
Signature Verified Successfully

Dupe https://stackoverflow.com/questions/57929493/openssl-digitally-sign-digest-only/ https://stackoverflow.com/questions/9380856/different-signatures-when-using-c-routines-and-openssl-dgst-rsautl-commands https://stackoverflow.com/questions/38767660/multiple-openssl-rsa-signing-methods-produce-different-results https://stackoverflow.com/questions/9951559/difference-between-openssl-rsautl-and-dgst https://stackoverflow.com/questions/13419201/why-are-the-rsa-sha256-signatures-i-generate-with-openssl-and-java-different — dave_thompson_085, May 21 '20 at 22:27
also cross https://security.stackexchange.com/questions/157496/ and https://https://crypto.stackexchange.com/questions/45069/ https://crypto.stackexchange.com/questions/27892/ — dave_thompson_085, May 21 '20 at 22:27
@dave_thompson_085 - First, I understand that the two tools produce different data. Second, your first dupe is comparing rsautl and dgst, not dgst and pkeyutl. [first suggested dupe](https://stackoverflow.com/questions/57929493/openssl-digitally-sign-digest-only/). Third [second suggested dupe](https://stackoverflow.com/questions/9380856/different-signatures-when-using-c-routines-and-openssl-dgst-rsautl-commands) was resolved with pkeyutl. #3, #4, #5 dgst and rsautl again. #6, #7, is 404, #8 rsautl again. I understand they hash differently. Which one is the proper one? That is the question. — seanlum, May 21 '20 at 22:49
@dave_thompson_085 yes, I understand there is a difference between "openssl dgst -sha256" and "openssl pkeyutl -pkeyopt digest:sha256", they are not the same. — seanlum, May 21 '20 at 22:52

score 1 · Answer 1 · edited Oct 07 '21 at 11:21

Okay, the first time I missed that you had omitted any hashing in the second method. Most people stumble on the difference between correct hash-and-sign with dgst -sign versus slightly incorrect hash-and-sign with dgst and rsautl or pkeyutl, not the totally incorrect sign-ONLY you did.

So the answer is: pkeyutl ALONE is COMPLETELY WRONG. (So is rsautl alone.)

The RSA signing primitive by itself is limited to small amounts of data, depending mostly on the key size and nowadays about 240 bytes. Most applications like documents, email, code, and communications (SSH, SSL/TLS) need to handle more data than this, so we always hash the data first and then RSA-sign the hash -- plus padding, which is where the issue usually comes in, see below. This is also true for DSA and ECDSA, and one variant of EdDSA, though the preferred variant of EdDSA uses a different solution.

See the current version of the PKCS1 standard (or any of its predecessors) and note 8.2.1 combined with 9.2 gives a four-step process: hash the data, 'encode' the hash in ASN.1 which is effectively adding a prefix (padding) as shown in 'Notes', explicitly pad the encoded hash with another padding (block type 1), and finally compute the RSA primitive r^d mod n. (Verification in 8.2.2 does the same but reverses the RSA primitive.)

openssl dgst -$hash -sign pubkey does the combined process correctly.

openssl dgst -$hash -binary | openssl pkeyutl -sign -pkeyopt digest:$hash is also correct (though more complicated).

openssl dgst -$hash -binary | openssl pkeyutl -sign WITHOUT -pkeyopt OR openssl dgst -$hash -binary | openssl rsautl -sign are wrong, but subtly so; they do hash-pad-sign, which seems correct, but they don't do the ASN.1 part of the padding. This is the difference explained in the numerous previous Qs I thought were duplicates.

Variations that do effectively the same thing using a temporary file or other medium instead of the pipe are equally correct or incorrect; this allows separating the hash part of the operation from the other parts, which can be useful if the data is on a different system or device than the key (or vice versa).

openssl pkeyutl -sign without a preceding hashing step is blatantly wrong.

I'm not sure quite what you are asking about 'RSA certificates' and 'X.509 certificates'. Both of the file formats you used -- PKCS8 for privatekey, and what OpenSSL calls PUBKEY for publickey -- are keys and not any kind of certificates, even though you misleadingly gave them names beginning with cert. Both of these file formats are 'generic' -- they support multiple algorithms, including RSA and many others. An X.509 certificate contains a publickey and MUCH other data, and is a different generic format supporting RSA and many other algorithms. Thus you can have a privatekey (PKCS8) file which contains an RSA privatekey and is often called simply an RSA privatekey file, a publickey file which contains an RSA publickey and is often called simply an RSA publickey file, and/or a (X.509) certificate file that contains an RSA publickey and is often called simply an RSA certificate.

dgst -sign/verify uses ONLY a privatekey or publickey file respectively, never a certificate. It supports multiple algorithms, not only RSA.

pkeyutl -sign uses only a privatekey file, but -verify can use a privatekey file, publickey file, or certificate file, with -pubin or -certin for the latter two. It supports multiple algorithms. rsautl accepts the same file formats, but only supports RSA.

Thank you very much for the detailed explanation. Especially regarding the application of the PKCS8 privatekey, RSA publickey, and X.509 certificate. Along with the explanation about usage of padding. My apologies for the incorrect terminology and references. — seanlum, May 23 '20 at 21:49

OpenSSL - Proper RSA Signature Generation and Verification

1 Answers1