OpenSSL and MS CryptoAPI: different digital signatures

Question

I generated X509 certificate with private key using makecert utility

makecert -n "CN=RootCATest" -r -sv RootCATest.pvk RootCATest.cer 
makecert -sk MyKeyName -iv RootCATest.pvk -n "CN=tempCert" -ic RootCATest.cer -sr currentuser -ss my -sky signature —pe

Then I converted RootCATest.pvk to RootCATest.pem with OpenSSL. And I extracted public key: pubRootCATest.pem

I have small file called 'msg'. And I sign this file using SHA1.

openssl dgst -sha1 -sign c:\RootCATest.pem -out c:\openssl c:\msg

Then I want to obtain the same digital signature using MS CryptoAPI.

Here is my code (Note: this is the code to understand concepts so I don't free allocated memory)

void SwapBytes(BYTE *pv, int n)
{
    BYTE *p = pv;
    int lo, hi;
    for(lo=0, hi=n-1; hi>lo; lo++, hi--)
    {
        BYTE tmp=p[lo];
        p[lo] = p[hi];
        p[hi] = tmp;
    }
}

void sign()
{
    FILE *file;
    BYTE *msg;
    int msg_size;

    HCRYPTPROV hProv;
    HCERTSTORE hStore;
    PCCERT_CONTEXT pCert;
    DWORD dwKeySpec;
    BOOL fCallerFreeProv;
    BYTE  *pSignature;
    DWORD sigLen;

    // Read message bytes from file
    file = fopen("c:\\msg", "r");
    fseek(file, 0, SEEK_END);
    msg_size = ftell(file);
    fseek(file, 0, SEEK_SET);
    msg = new BYTE[msg_size];
    fread(msg, sizeof(BYTE), msg_size, file);
    fclose(file);

    hStore = CertOpenSystemStore(NULL, "My");
    pCert = CryptUIDlgSelectCertificateFromStore(hStore, NULL, NULL, NULL, 0, 0, NULL);
    CryptAcquireCertificatePrivateKey(pCert, CRYPT_ACQUIRE_COMPARE_KEY_FLAG, NULL, &hProv, &dwKeySpec, &fCallerFreeProv);
    PrintCryptoProviderName(hProv); // prints Microsoft Strong Cryptographic Provider

    ALG_ID hashAlgId = CALG_SHA1;
    HCRYPTHASH hHash;
    CryptCreateHash(hProv, CALG_SHA1, 0, 0, &hHash);
    CryptHashData(hHash, msg, msg_size, 0);

    CryptSignHash(hHash, dwKeySpec, NULL, 0, NULL, &sigLen);
    pSignature = new BYTE[sigLen];
    CryptSignHash(hHash, dwKeySpec, NULL, CRYPT_NOHASHOID, pSignature, &sigLen);

    SwapBytes(pSignature, sigLen); // Here i reverse byte order as I read that MS CryptoAPI uses reversed byte order

    // Write signature bytes to file
    file = fopen("c:\\CryptSignHash", "w");
    fwrite(pSignature, sizeof(BYTE), sigLen, file);
    fclose(file);
}

As output I get the signature absolutely different from the signature made by OpenSSL. How can I obtain the same signature?

As I consider there are some moments to pay attention:

My msg_size is the same as file size. So it is the number of bytes to sign. On some sites I saw recommendations to add a null byte to byte array. Do I really need it in such a case?
The flag CRYPT_NOHASHOID. Without it I get the signature of size 130 bytes, when the signature made by OpenSSL is 128 bytes. So I think CRYPT_NOHASHOID should be there.
SwapBytes(...) I tried with it and without it. And in both cases I have signatures absolutely different to OpenSSL signature.

There are many different formats for both the data going into the signature as well as the signature itself. This is the part you'll have to get right. — President James K. Polk, Dec 20 '11 at 22:09
Now, use RSA_verify to verify this signature and if it is verifiable, then it is same. — doptimusprime, Apr 11 '13 at 05:23

score 0 · Answer 1 · edited Jun 09 '15 at 03:19

0

How can I obtain the same signature?

Most digital signature algorithms - including RSA, which I suppose you have used here, are non-deterministic. Try signing the same file twice with the same program, and you will get different outputs.

This means, running the same algorithm twice with the same input will give you different signatures. This is not a problem, as long as the verification algorithm still manages to accept all signatures generated by the signing algorithm (with the fitting key).

This non-determinism is often actually necessary for the security of the signature scheme.

To see if your two signature algorithms are actually compatible, try to verify the OpenSSL signature with the MS Crypto API, and to verify the MS Crypto signature with OpenSSL. (Then modify the file by one byte and check that they don't verify anymore.)

edited Jun 09 '15 at 03:19

jww

97,681
90
411
885

answered Dec 23 '11 at 23:05

Paŭlo Ebermann

73,284
20
146
210

Paulo, I managed to obtain the same signature. But I changed the task. – Stanislav Dec 27 '11 at 09:46
2

I generated key and signature with MS CryptoAPI and validate with OpenSSL. And vice versa. And the signatures were unique! So i suppose i have deal with deterministic RSA in my case. – Stanislav Dec 27 '11 at 13:38
If same data is signed by same private key, signature will always be same as padding is not random. – doptimusprime Apr 11 '13 at 05:22
1

Not sure why you were down voted.... I'm fairly certain everything you said was spot-on. – jww Jun 09 '15 at 03:12
RSA is *not* non-deterministic. https://en.wikipedia.org/wiki/RSA_(cryptosystem) – Joe Strommen Oct 01 '15 at 13:41
1

@JoeStrommen Textbook RSA signing is deterministic, but there are also non-deterministic variants around (RSA-PSS, for example, mentioned under "Padding" in the Wikipedia article linked by you). I guess (from the comments of Stanislav) that the one used here is a deterministic one, then I can't really help anymore. (Though I still don't see the point on getting "exactly the same signature".) – Paŭlo Ebermann Oct 05 '15 at 08:45

OpenSSL and MS CryptoAPI: different digital signatures

1 Answers1

Linked