OpenSSL PEM_read_RSAPublicKey and PEM_read_RSA_PUBKEY crashes the program

Question

I have a public key in the form of

-----BEGIN PUBLIC KEY-----
MIIB...
-----END PUBLIC KEY-----

that was created using OpenSSL. I attempt to load it into memory like such:

FILE* file = fopen("public.pem","r");
if (file != nullptr)
{
   //This can be changed to this for the same results:
   //RSA* rsa = PEM_read_RSAPublicKey(file, nullptr, nullptr, nullptr);
   RSA* rsa = PEM_read_RSA_PUBKEY(file, nullptr, nullptr, nullptr);
   //etc...
}

Whenever it gets to the RSA* rsa =... line the program crashes. I have seen the other post for this and (as you can see) I have tried it the solution and it doesn't work.

I have tried using the BIO versions and the program does not crash but I do get the error "no start line".

Any ideas as to what can be causing this?

I think that you need to point to valid structures where data will be read to instead of using nullptr. — Mr. Developerdude, Dec 11 '16 at 22:27
Maybe the method PEM_read_RSA_PUBKEY shouldn't be given `nullptr`s? — Top Sekret, Dec 11 '16 at 22:27
I have tried: if (!PEM_read_RSA_PUBKEY(file, &rsa, nullptr, nullptr)) (where rsa was initialized using RSA_new()) and it still crashes. The last two fields are just password fields if you use a password. The second to last is a callback function pointer to find the password and the last one is a void* that is pointing to a char*. Mine does have a password but using (void*)password for the last parameter does not help. — Ricky L., Dec 11 '16 at 22:35

score 1 · Answer 1 · edited May 23 '17 at 12:13

RSA* rsa = PEM_read_RSA_PUBKEY(file, nullptr, nullptr, nullptr);

The second argument cannot be NULL. That's causing your crash. In addition, its a pointer-to-a-pointer (double indirection). Also see the PEM_read_RSA_PUBKEY man pages.

Maybe you should use something like:

RSA* t = RSA_new();
RSA* rsa = PEM_read_RSA_PUBKEY(file, &t, nullptr, nullptr);

The data may be malformed, and it could be causing an unexpected crash. If you provide real data, then we can usually cross-check it. In the absence of real data, we can't really say.

You might also want to visit How to generate RSA private key using openssl?. It shows you several RSA output formats. It also shows you how to manage OpenSSL resources using C++.

I tried your option and it still crashed. I switched to using file BIOs instead and that worked. I've had issues with OpenSSL and FILE before, so idk what's going on. Thanks for the input :) — Ricky L., Dec 16 '16 at 01:31

OpenSSL PEM_read_RSAPublicKey and PEM_read_RSA_PUBKEY crashes the program

1 Answers1