rsa_public_encrypt returns -1, error 0x0406B07A

Question

I'm trying to encrypt data using RSA_public_encrypt but it doesn't seem to work (retEnc is always -1). I also tried to find more information about the error using ERR_get_error and ERR_error_string.

This is the code :

RSA *rsaPkey = NULL;

FILE *pemFile;
fopen_s(&pemFile, filename.c_str(), "r");
rsaPkey         = PEM_read_RSA_PUBKEY(pemFile, &rsaPkey, NULL, NULL);
fclose(pemFile);

if(rsaPkey == NULL)
    throw "Error pubkey file";

int size = RSA_size(rsaPkey);
unsigned char *encrypted;
encrypted = new unsigned char[size];

string instr  = "test";
int length = instr.length();
unsigned char *in = (unsigned char *)(instr.c_str());

unsigned long errorTrack = ERR_get_error() ;

int retEnc = RSA_public_encrypt(length, in, (unsigned char *)encrypted, rsaPkey, RSA_NO_PADDING);
errorTrack = ERR_get_error() ;
char *errorChar = new char[120];
errorChar = ERR_error_string(errorTrack, errorChar);

ERR_error_string gives me error:0406B07A:lib(4):func(107):reason(122)

How can I find more details about this, where can i find the library 4 and function 107 ?

When I try to encrypt using openssl cli and the same public key file, the encryption works fine.

You might also be interested in [Library Initialization](https://wiki.openssl.org/index.php/Library_Initialization) on the OpenSSL wiki. — jww, Mar 02 '18 at 12:59

jww · Accepted Answer · 2018-03-02T13:17:18.593

ERR_error_string gives me error:0406B07A:lib(4):func(107):reason(122)
How can I find more details about this, where can i find the library 4 and function 107 ?

I find the easiest way to learn more from an OpenSSL error code is:

$ openssl errstr 0406B07A
error:0406B07A:rsa routines:RSA_padding_add_none:data too small for key size

char *errorChar = new char[120];
errorChar = ERR_error_string(errorTrack, errorChar);

Also, from the ERR_error_string man page:

ERR_error_string() generates a human-readable string representing the error code e, and places it at buf. buf must be at least 256 bytes long. If buf is NULL, the error string is placed in a static buffer. Note that this function is not thread-safe and does no checks on the size of the buffer; use ERR_error_string_n() instead.

Since you are using C++, something like this may be easier:

std::string errorMsg;
errorMsg.resize(256);

(void)ERR_error_string(errorTrack, &errorMsg[0]);

Above, you are using a std::string to manage resources. To get the non-const pointer, you take the address of the first element.

If you want, you can properly size the errorMsg with:

(void)ERR_error_string(errorTrack, &errorMsg[0]);
errorMsg.resize(std::strlen(errorMsg.c_str()));

Here is another trick that might make C++ a little easier to use.

typedef unsigned char byte;
...

std::string encrypted;
int size = RSA_size(rsaPkey);

if (size < 0)
    throw std::runtime_error("RSA_size failed");

// Resize to the maximum size
encrypted.resize(size);
...

int retEnc = RSA_public_encrypt(length, in, (byte*)&encrypted[0], rsaPkey, RSA_NO_PADDING);

if (retEnc < 0)
    throw std::runtime_error("RSA_public_encrypt failed");

// Resize the final string now that the size is known
encrypted.resize(retEnc );

Above, you are using a std::string to manage resources. To get the non-const pointer, you take the address of the first element.

Also, NO_PADDING is usually a bad idea. You usually want OAEP padding. See the notes in RSA_public_encrypt man page on how padding affects the maximum size.

C++ can make it easier to use OpenSSL. You can avoid explicit calls to functions like EVP_CIPHER_CTX_free by using unique_ptr. See EVP Symmetric Encryption and Decryption | C++ Programs on the OpenSSL wiki, unique_ptr and OpenSSL's STACK_OF(X509)*, How to get PKCS7_sign result into a char * or std::string, etc.

In your case, it looks like these would be helpful to manage resources:

using FILE_ptr = std::unique_ptr<FILE, decltype(&::fclose)>;
using RSA_ptr = std::unique_ptr<RSA, decltype(&::RSA_free)>;

rsa_public_encrypt returns -1, error 0x0406B07A

1 Answers1