1

Trying to wrap my head around signing and use/test various options.

I can sign using this command:

openssl dgst -sha256 -sign private_key.pem -binary -out sig_file data_file

But the documentation seems to say that I can also use this method

openssl dgst -sha256 -binary data_file > hash_file
openssl rsautl -sign -inkey private_key.pem -keyform PEM -in hash_file > sig_file2

But the signatures are different when I'd expect them to be identical. Either I missed something in the options or something else is wrong in my assumptions.

The real question from this issue: Is there a way to sign using command line options given that I already have the hash value and produce a signature that is identical to the first command above.

To add more to this, I can reproduce the first command easily in code and it matches the first command above meaning that I can sign with the hash value calculated first.

mdctx = EVP_MD_CTX_create();
EVP_DigestInit_ex(mdctx, EVP_sha256(), NULL);
EVP_DigestUpdate(mdctx, data, len);
EVP_DigestFinal_ex(mdctx, hash, &s);
EVP_MD_CTX_destroy(mdctx);

kfile = fopen64(key_file, "r");
key = PEM_read_RSAPrivateKey(kfile, NULL, NULL, NULL);
fclose(kfile);

*sig = malloc(RSA_size(key));
RSA_sign(NID_sha256, hash, hlen, *sig, siglen, key);
jww
  • 97,681
  • 90
  • 411
  • 885
David
  • 3,324
  • 2
  • 27
  • 31

1 Answers1

3

Dupe: Difference between openSSL rsautl and dgst
Closely related:
Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?
Different signatures when using C routines and openssl dgst, rsautl commands
Signing 20-byte message with 256-bit RSA key working with openssl.exe but not in code
Crossdupe: https://superuser.com/questions/943972/what-is-the-difference-between-openssl-pkeyutl-sign-and-openssl-rsautl-sign

TLDR: dgst -sign for RSA does the full RSASSA-PKCS1-v1_5: hash the data, encode the hash in ASN.1, pad the result, and modexp d. rsautl -sign does only the last two and dgst by itself only the first, thus skipping the encode producing a different and nonstandard signature. dgst (or your own hash) then pkeyutl -sign with an RSA key and -pkeyopt digest:name_of_digest (important!) also works and answers your real question.

dave_thompson_085
  • 34,712
  • 6
  • 50
  • 70