3

I am working on C RPC client. I need to use OpenSSL for signing mime data, and send the public key and signature to server. Java will try to verify the signature using public key.

Problem is, JAVA failed to verify OpenSSL generated signature but could verify JAVA produced signature. Any idea of how to fix my c client code?

c client code:

#define LENGTH 256
FILE *fp=fopen("ssl.key");
unsigned int uDigestLen = 32;
EVP_PKEY *privkey;
privkey = EVP_PKEY_new();
PEM_read_PrivateKey( fp, &privkey, NULL, NULL);

pRsaKey = EVP_PKEY_get1_RSA(privkey);

if(RSA_check_key(pRsaKey)) {
    printf("RSA key is valid.\n");
}

EVP_MD_CTX_init(&md_ctx);
EVP_DigestInit(&md_ctx, EVP_sha256());
EVP_DigestUpdate(&md_ctx, (const void*) szMessage, strlen(szMessage));
EVP_DigestFinal(&md_ctx, pDigest, &uDigestLen);
EVP_MD_CTX_cleanup(&md_ctx);

// also tried RSA_padding_add_PKCS1_PSS but no luck
status = RSA_padding_add_PKCS1_PSS_mgf1(pRsaKey, EM, pDigest, EVP_sha256(), EVP_sha256(), 20 /* fixed salt length! (tried -2 w/o success)*/);
if (!status)
{
    printf("RSA_padding_add_PKCS1_PSS failed with error %s\n", ERR_error_string(ERR_get_error(), NULL));
}

status = RSA_private_encrypt(LENGTH, EM, pSignature, pRsaKey, RSA_PKCS1_PADDING);
if (status == -1)
{
    printf("RSA_private_encrypt failed with error %s\n", ERR_error_string(ERR_get_error(), NULL));
}

// I send base64 encoded string as well as public key to server, java failed to verify.
cout << "hashedChars: " << base64_encode(pSignature, LENGTH);

JAVA code:

Signature publicSignature = Signature.getInstance("SHA256withRSA/PSS", "BC");
publicSignature.initVerify(publicKey);
publicSignature.update(text.getBytes(StandardCharsets.UTF_8));

// always fail!!
byte[] signatureBytes = Base64.getDecoder().decode("base64 encoded  from c client");
System.out.println("verified result:" + publicSignature.verify(signatureBytes));
jww
  • 97,681
  • 90
  • 411
  • 885
Yuhui Liu
  • 97
  • 1
  • 6
  • I am very new to this domain and don't quite understand how this may affect my c client code. Given openSSL's structure doesn't meet JAVA's expectation, should I switch to crypto++ for RSA/PSS signing? Is crypto's signature compatible with JAVA? – Yuhui Liu Feb 15 '18 at 09:43
  • I think this is your duplicate: [How to convert digital signature to ASN1?](https://stackoverflow.com/q/40247454/608639) – jww Feb 15 '18 at 10:32
  • Here's another very relevant question: [Sign with ECDSA and openssl, do NOT put message](https://stackoverflow.com/q/17316178/608639). – jww Feb 15 '18 at 16:23

1 Answers1

2

I guess there are two things to fix here:

  1. set MGF1 to use SHA-256 explicitly;
  2. set the salt size, where OpenSSL uses the largest salt size possible.

See for more info the old discussion here.

Maarten Bodewes
  • 90,524
  • 13
  • 150
  • 263
  • If you post a signature and public key I'm of course willing to try. Running / compiling the C code is currently not an option for me. – Maarten Bodewes Feb 18 '18 at 16:48
  • 1
    I mananged to figure out the magic. We need to use PSSParameterSpec spec1 = new PSSParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-256"), 20, _trailer); and the trailer value is 0xBC for open ssl. Someone said we could use max salt length to replace '20' but it didn't work for me. – Yuhui Liu Feb 28 '18 at 18:44