C++ Public Key Encryption with RSA_public_encrypt

Question

I'm currently struggeling with the public Key Encryption in C++. I am using openssl but the generated string is always broken. My current solution is the following:

string encryptWithPublicKey(const char* pData, int iLen) { // pData: String to encrypt, iLen: length

        char chPublicKey[] = "---Public Key-- ...";

        BIO* bio = BIO_new_mem_buf(chPublicKey, -1);

        RSA* rsa = PEM_read_bio_RSA_PUBKEY(bio, NULL, NULL, NULL);
    

        int nLen = RSA_size(rsa);
        char* pEncode = (char*)malloc(nLen + 1);
        int rc = RSA_public_encrypt(iLen, (unsigned char*)pData, (unsigned char*)pEncode, rsa, RSA_NO_PADDING);

        cout << "From: " << pData << " To: '" << pEncode << "'";

        RSA_free(rsa);
        CRYPTO_cleanup_all_ex_data();
        std::string strName = std::string(pEncode);

        return strName;
}

The output of the encryption is always in that format: "═════════════════════════════════════════════════════════════════²²²²☻"

No idea why the output has that format ^^

Maybe someone has an idea how to use the encryption in a better.

Thanks Kurt

the encrypted data is binary, you probably want to base64 encode or something before printing — Alan Birtles, Sep 28 '20 at 15:05
You forgot to `free pEncode` . Anyway in C++ you should use `new` and `delete`, or just `std::string` with `c_str` method if you need to access raw array. — pptaszni, Sep 28 '20 at 15:09
[`RSA_public_encrypt`](https://www.openssl.org/docs/man1.1.1/man3/RSA_public_encrypt.html) is not for encrypting arbitrary data, it is for encrypting *keys*. In case of `RSA_NO_PADDING` the input must be exactly the size of `RSA_size(rsa)` (i.e. the *key* size). — rustyx, Sep 28 '20 at 15:15
The normal process is to exchange temporary AES keys via RSA and then use AES for encryption. Though I would just use the tried and true HTTPS and be done with it. — rustyx, Sep 28 '20 at 15:28

score 1 · Answer 1 · answered Sep 29 '20 at 12:25

Encrypted data is binary. You shouldn't print binary data. You should encode it as base64 or hex before printing.
════════════ are 0xCC bytes, magic fill bytes for uninitialized memory. It seems nothing was writted to the output, which means the call to RSA_public_encrypt failed.

You are not checking the return value rc.

int rc = RSA_public_encrypt(iLen, (unsigned char*)pData, (unsigned char*)pEncode, rsa, RSA_NO_PADDING);

Add some error reporting to see the exact error:

if (rc < 0) {
    char buf[128];
    cerr << "RSA_public_encrypt: " << ERR_error_string(ERR_get_error(), buf) << endl;
}

It will probably print something like RSA_padding_add_none:data too small for key size.

That's because RSA_public_encrypt encrypts exactly RSA_size(rsa) bytes (the size of the RSA key), and is meant primarily for encrypting symmetric keys, which are in turn used for encryption.

I use public/private key for symmetric key transfer

Good. In that case the key should fit and you just need to pad it to RSA_size(rsa) bytes. Or better yet, use some good padding like RSA_PKCS1_OAEP_PADDING instead of RSA_NO_PADDING.

Note that std::string(pEncode) expects a C-string input. You need std::string(pEncode, rc) in order to store binary data inside std::string. Or better yet, use base64 (see EVP_EncodeBlock).

C++ Public Key Encryption with RSA_public_encrypt

1 Answers1