9

I'm trying to develop a license verification solution. Licenses are encoded on server using OpenSSL's RSA_private_encrypt function.

For Mac OX X I use RSA_public_decrypt and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with OpenSSL or other lib AND I have to use MS Crypto API.

I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends. I'm aware that I need to reverse byte order with CAPI so this might not be the issue.

I have tried everything, including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load the blob with different params, but still no luck.

It always ends up with GetLastError() == CRYPT_E_ASN1_BADTAG, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.

Here is the OS X code based on OpenSSL:

void cr_license_init(const char* lic) {
    __cr_license_ = lic;
    unsigned char lic_encoded[CR_LIC_LEN];

    BIO* b64 = BIO_new(BIO_f_base64());
    BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
    licIn = BIO_push(b64, licIn);

    if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {

        const unsigned char* key_data = license_pub_der;
        RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der));

        if(r != NULL) {
            if(__cr_license_data_ != NULL) {
                free((void*)__cr_license_data_);
            }
            __cr_license_data_ = malloc(CR_LIC_LEN);
            if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
    (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &lt= 0) {
                free((void*)__cr_license_data_);
                __cr_license_data_ = NULL;
            }
            RSA_free(r);
        }
    }
    BIO_free_all(licIn);
}

This part of code on windows works well, so I assume public key is not an issue.

__cr_license_ = lic;
unsigned char lic_encoded[CR_LIC_LEN];

DWORD dwSize;
if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) {
HCRYPTPROV hProv;
if(CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
    PCERT_PUBLIC_KEY_INFO pki = NULL;
    DWORD dwKeySize;
    if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) {
        HCRYPTKEY hKey = 0;
        if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) {

But after that anything I try to do with message leads to CRYPT_E_ASN1_BADTAG. I tried CryptMsgOpenToDecode with CryptMsgUpdate, CryptDecodeObject, CryptVerifyMessageSignatureWithKey - nothing works.

Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentioned. Does anyone has experience working with pkcs1 format importing/converting/etc with MS CAPI?

Any help or even a clue is appreciated a lot! Thanks in advance!

jww
  • 97,681
  • 90
  • 411
  • 885
strannik
  • 1,595
  • 1
  • 13
  • 22
  • You might want to include some code in the question. – Daniel Roethlisberger Jan 25 '13 at 18:42
  • 1
    also show header of keys and encoded files. That error is for invalid ASN tag header – Vahid Farahmand Jan 25 '13 at 20:52
  • Does this help? http://stackoverflow.com/q/7573754/645583 – pcunite Jan 25 '13 at 22:22
  • Yeah, I have seen this thread. The problem in this case is not the key. I can successfully import the key with CryptImportPublicKeyInfo, so I assume the key is not the case. It seems that the problem is with encoded BLOB. I believe that the BLOB is not ASN1 compatible and should be converted somehow. – strannik Jan 25 '13 at 22:41
  • Did you check the comment below this article? http://msdn.microsoft.com/en-us/library/windows/desktop/aa381092(v=vs.85).aspx – Maarten Bodewes Jan 25 '13 at 22:45
  • Thanks @owlstead, I have seen this comment of course. I doubt this is the case, because I can import the public key into the context with CryptImportPublicKeyInfo successfully. – strannik Jan 25 '13 at 23:00
  • 1
    @Daniel I was hoping for the Windows code which threw the `CRYPT_E_ASN1_BADTAG` error you are refering to. – Daniel Roethlisberger Jan 25 '13 at 23:05
  • @DanielRoethlisberger, sorry I have no Windows code at home, but I will definitely add some tomorrow at office. – strannik Jan 25 '13 at 23:08
  • Daniel, would you know the padding mechanism used by `CryptVerifyMessageSignatureWithKey`? For some reason it is not specified by the API (not that I'm surprised, Microsoft regularly underspecifies) – Maarten Bodewes Jan 25 '13 at 23:12
  • @owlstead, Microsoft tells you can choose one: X509_ASN_ENCODING | PKCS_7_ASN_ENCODING. So BLOB certainly must be ASN1 formatted and seems that it works with PKCS padding. – strannik Jan 25 '13 at 23:15
  • PKCS#7 or CMS is a high level cryptographic message syntax. OpenSSL code above seems to use a low level PKCS#1 signature. This signature format can be *part of* PKCS#7 structure, but is certainly not the same. PKCS stands for public-key cryptography standards and requires a number such as #1 #7 to point to a specific standard (and this should hopefully be followed by a version and the name of the specific message syntax or algorithm). PKCS in itself doesn't define anything. This might answer your question. Note that CMS is specified using ASN.1 BER notation.... – Maarten Bodewes Jan 25 '13 at 23:21
  • X509_ASN_ENCODING is for certificates by the way, they are also specified by ASN.1 BER notation. Certificates are signed in much the same way as CMS, basically they are a container format that is signed, again, the signature itself is likely to be PKCS#1 (using the v1.5 signature format). – Maarten Bodewes Jan 25 '13 at 23:25
  • @owlstead, OK, got it, thanks for so detailed explanation! Any suggestions how could PKCS1 be converted to PKCS7? – strannik Jan 25 '13 at 23:26
  • Conversion may not be possible. PKCS#7 may use a PKCS#1 signature format, but it is normally not directly calculated over the data itself. Either let the server create PKCS7 (it is OpenSSL functionality) or try and find a function that will verify PKCS#1 signatures. – Maarten Bodewes Jan 25 '13 at 23:31
  • @DanielRoethlisberger, I posted the windows code part. – strannik Jan 26 '13 at 14:47
  • @Daniel: Two question. I saw that you was asking this 3 month ago and just created a bounty. Any progress on this. Can you please post public, private key and encrypted license (for testing). So, I can take a look at it and try to write a code which will decrypt it on Windows. – Victor Ronin Apr 21 '13 at 20:58
  • @Daniel: One more question. Where this requirement of tiny code come from? Is this for a driver or something like it? – Victor Ronin Apr 21 '13 at 20:59
  • Also see [OpenSSL and MS CryptoAPI: different digital signatures](http://stackoverflow.com/q/8572986). – jww Jun 09 '15 at 03:16

3 Answers3

4

You are mixing higher and lower level signature formats. OpenSSL asumes PKCS#1 v1.5 signatures by default, which contains of only the signature data. Windows seems to asume PKCS#7 containers. These may contain a PKCS#1 v1.5, but those and other data are wrapped using ASN.1 BER tag/length format. If the Microsoft API tries to decode this it will assume that the raw signature is the container format, and decoding will fail.

Maarten Bodewes
  • 90,524
  • 13
  • 150
  • 263
1

Unless this is so obvious that you've tried but omitted listing it or I misunderstand your question otherwise, I think you should be using CryptDecrypt to decrypt the license, not the functions you mention in the question. Note that since you seem to be using OpenSSL with PKCS#1 v1.5 padding and CryptoAPI does not seem to support that (haven't tested, but specs only list PKCS#1 v2 OAEP), you will probably have to use CRYPT_DECRYPT_RSA_NO_PADDING_CHECK and verify and remove the PKCS#1 v1.5 padding manually after decryption.

Daniel Roethlisberger
  • 6,958
  • 2
  • 41
  • 59
  • Isn't CryptDecrypt intended to be used with the private key? – strannik Jan 26 '13 at 15:07
  • As I thought it returns NTE_NO_KEY when used with public key. – strannik Jan 26 '13 at 15:16
  • Have you tried or do you just think it does? I honestly don't know, but all the other functions operate on higher level cryptographical containers, not raw RSA blocks. If it does not work, you might either be able to trick it into believing the public key is a private key by flipping some bit (not sure how the public key is marked as such), or trick it using CryptEncrypt without any padding (not specifying CRYPT_OAEP), since if it does not do any padding, encrypt and decrypt is the equivalent operation. – Daniel Roethlisberger Jan 26 '13 at 15:20
  • Yeah, I just tried. It does not work. I'm not sure if it is possible (and how) to trick it to believe that it's a private key... Basically I think this might not be possible, because encrypting with private key (also known as "signing") should differ a lot from encrypting with public key. – strannik Jan 26 '13 at 15:28
  • The encrypt and decrypt operations only differ in adding the padding / verifying&removing padding. On the raw RSA level, the operations are identical, just with a different key. Also, the raw signing/verifying operations only differ from encryption/decryption in the different padding, the actual RSA operations on the bit level are identical. – Daniel Roethlisberger Jan 26 '13 at 15:29
0

OpenSSL exports keys with extra header which is not expected by CryptoAPI.

Header for private key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   630|      3| SEQUENCE : 
     4|     1|      1|    INTEGER : 0
     7|    13|      1|    SEQUENCE : 
     9|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    20|     0|      1|       NULL : 
    22|   608|      3|    OCTET STRING : 
                             ... actual key data go here ...

Header for public key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   159|      2| SEQUENCE : 
     3|    13|      1|    SEQUENCE : 
     5|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    16|     0|      1|       NULL : 
    18|   141|      2|    BIT STRING UnusedBits:0 : 
                              ... actual key data go here ...

These headers are what causes CryptDecodeObjectEx to choke. It expects RAW key data, without any header.

So, basically, you need:

  1. (Optional) Convert .PEM to .DER with CryptStringToBinary.
  2. Check if DER starts with the above mentioned headers. For that you need to read ASN.1-encoded data.
  3. (Optional) Skip the above mentioned header and seek directly to key's data (starts with SEQUENCE which includes 2 INTEGER for public key or 9 INTEGER for private key).
  4. Feed the result to CryptDecodeObjectEx(X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB/PKCS_RSA_PRIVATE_KEY).
  5. Import keys with CryptImportKey.
Alex
  • 5,477
  • 2
  • 36
  • 56