rsa_private_decrypt returns -1, error 0x0306B067

Question

I want to test a RSA key pair using OpenSSL in C++. I encrypt a text and decrypt it immediately. The encryption is done successfully but the decryption return -1 always. The error number is:

0306B067:lib(3):func(107):reason(103)

I couldn't find out the reason... the code is:

char* text = "hello!!!";
unsigned char * cipher ;
unsigned char * decipher ;
int size = RSA_size(prvKey);
cipher = (unsigned char *)malloc(size);
decipher = (unsigned char *)malloc(size);
int cipherres = RSA_public_encrypt(size - 11/*strlen(text)*/,(unsigned char*)text,cipher,pubKey,RSA_PKCS1_PADDING);

int decipherres = RSA_private_decrypt(size,cipher,decipher,prvKey,RSA_PKCS1_PADDING);
if (decipherres == -1)

RSA key pair was generated befor using RSA_generate_key_ex() function and it's key pair are in pubKey and prvKey.

Answer 1

I don't like the look of this line:

int cipherres = RSA_public_encrypt(size - 11/strlen(text)/,...

You should be using the length of text, which it seems you were doing previously. I don't know where you get the 11 from but magic numbers in code smell bad.

Try:

int cipherres = RSA_public_encrypt(strlen(text),...

Also, referring to the documentation for RSA_public_encrypt here, the following line looks wrong:

int size = RSA_size(prvKey);

Are prvKey and pubKey the same size by definition? If not, try using the public key to get size before the encrypt:

int size = RSA_size(pubKey);

You may then need to separately get the size of prvKey and malloc the decrypt buffer to that size.

Answer 2

There are several problems with your program. Some of them are pointed out by @Resource. Since you have this tagged as C++, here's the C++ way to do these things. Its the same basic program you wrote but with many of the kinks worked out.

The answer also uses How to generate RSA private key using OpenSSL? for the RSA keypair generation.

OpenSSL recommends you use the EVP interfaces instead of RSA_public_encrypt and RSA_private_decrypt. Also see EVP Asymmetric Encryption and Decryption on the OpenSSL wiki.

#include <openssl/bn.h>
#include <openssl/evp.h>
#include <openssl/err.h>
#include <openssl/rsa.h>

#include <string>
#include <memory>
#include <sstream>
#include <iostream>
#include <stdexcept>

#include <cassert>
#define ASSERT assert

using BN_ptr = std::unique_ptr<BIGNUM, decltype(&::BN_free)>;
using RSA_ptr = std::unique_ptr<RSA, decltype(&::RSA_free)>;

int main(int argc, char* argv[])
{   
    ////////////////////////////////////////////////////////////

    OpenSSL_add_all_algorithms();
    ERR_load_crypto_strings();

    ////////////////////////////////////////////////////////////

    RSA_ptr privKey(RSA_new(), ::RSA_free);
    RSA_ptr pubKey(RSA_new(), ::RSA_free);
    BN_ptr bn(BN_new(), ::BN_free);
    int rc;

    // Set exponent
    rc = BN_set_word(bn.get(), RSA_F4);
    ASSERT(rc == 1);
    if (rc != 1)
    {
        std::ostringstream msg;
        msg << "BN_set_word failed. Error 0x" << std::hex << ERR_get_error();
        throw std::runtime_error(msg.str());
    }

    // Generate private key
    rc = RSA_generate_key_ex(privKey.get(), 2048, bn.get(), NULL);
    ASSERT(rc == 1);
    if (rc != 1)
    {
        std::ostringstream msg;
        msg << "RSA_generate_key_ex failed. Error 0x" << std::hex << ERR_get_error();
        throw std::runtime_error(msg.str());
    }

    // Create a public key from private key
    pubKey.reset(RSAPublicKey_dup(privKey.get()));

    ////////////////////////////////////////////////////////////

    std::string text = "Yoda said, Do or do not. There is no try.";
    std::string cipher, decipher;

    int size = RSA_size(privKey.get());
    ASSERT(size >= 0);
    if (size < 0)
    {
        std::ostringstream msg;
        msg << "RSA_size failed. Error 0x" << std::hex << ERR_get_error();
        throw std::runtime_error(msg.str());
    }

    // 41 due to RSA_PKCS1_OAEP_PADDING
    ASSERT(text.length() + 41 < size);
    if (text.length() + 41 >= size)
    {
        std::ostringstream msg;
        msg << "Plain text length is too long for modulus and padding";
        throw std::runtime_error(msg.str());
    }

    // Resize to maximum size
    cipher.resize(size);
    decipher.resize(size);

    rc = RSA_public_encrypt(text.length(),(unsigned char*)&text[0],
            (unsigned char*)&cipher[0], pubKey.get(), RSA_PKCS1_OAEP_PADDING);
    ASSERT(rc >= 0);
    if (rc < 0)
    {
        std::ostringstream msg;
        msg << "RSA_public_encrypt failed. Error 0x" << std::hex << ERR_get_error();
        throw std::runtime_error(msg.str());
    }

    // Resize now that we know the size
    cipher.resize(size);

    rc = RSA_private_decrypt(cipher.length(), (unsigned char*)&cipher[0],
            (unsigned char*)&decipher[0], privKey.get(), RSA_PKCS1_OAEP_PADDING);
    ASSERT(rc >= 0);
    if (rc < 0)
    {
        std::ostringstream msg;
        msg << "RSA_private_decrypt failed. Error 0x" << std::hex << ERR_get_error();
        throw std::runtime_error(msg.str());
    }

    std::cout << "Message: " << text << std::endl;
    std::cout << "Recovered: " << decipher << std::endl;

    return 0;
}

The program only needs the crypto portion of the library. The program was built from the OpenSSL 1.0.2 source directory with:

$ g++ -std=c++11 -DNDEBUG -I ./include test.cxx ./libcrypto.a -ldl -o test.exe
$ ./test.exe
Message: Yoda said, Do or do not. There is no try.
Recovered: Yoda said, Do or do not. There is no try.

---

Regarding this:

0306B067:lib(3):func(107):reason(103)

Try:

$ openssl errstr 0306B067
error:0306B067:bignum routines:BN_div:div by zero

My guess is, there is something wrong with pubKey or privKey.