Segmentation fault with generating an RSA and saving in ASN.1/DER?

Question

#include <string.h>
#include <openssl/aes.h>
#include <openssl/rand.h>
#include <openssl/bio.h>
#include <openssl/rsa.h>
#include <openssl/evp.h>
#include <openssl/pem.h>

#define RSA_LEN 2048
#define RSA_FACTOR 65537

int genRSA2048(unsigned char **pub,unsigned int *pub_l,unsigned char **priv,unsigned int *priv_l){

    RSA *pRSA = NULL;
    pRSA = RSA_generate_key(RSA_LEN,RSA_FACTOR,NULL,NULL);

    if (pRSA){
        pub_l = malloc(sizeof(pub_l));
        *pub_l = i2d_RSAPublicKey(pRSA,pub);
        priv_l = malloc(sizeof(priv_l));
        *priv_l = i2d_RSAPrivateKey(pRSA,priv);
        return 0;
    } else {
        return 1;
    }
}

int main(){
    unsigned char *pub = NULL;
    unsigned int publ;
    unsigned char *priv = NULL;
    unsigned int privl;
    genRSA2048(&pub,&publ,&priv,&privl);


    RSA *privrsa = NULL;
    d2i_RSAPrivateKey(&privrsa,(const unsigned char **)&priv,privl);

    RSA *pubrsa = NULL;
    d2i_RSAPublicKey(&pubrsa,(const unsigned char **)&pub,publ);

    unsigned char * data ="01234567890123456789012345678912";
    unsigned char encrypted[256];
    unsigned char decrypted[32];

    int len = RSA_private_encrypt(32,data,encrypted,privrsa,RSA_PKCS1_PADDING);
    RSA_public_decrypt(len,encrypted,decrypted,pubrsa,RSA_PKCS1_PADDING);
}

I've tried to find the bug by checking with gdb but as being fairly new to C I haven't find any clue to tell me what is happening but I believe it's an allocation problem, however according to the d2i_RSAPrivateKey and similar, they're supposed to allocate the space by itself.

Any help would be greatly appreciated.

compiled as cc foo.c -lcrypto

This is the follow up of this question:

Generate RSA public/private key with OpenSSL?

As I reference I used @WhozCraig example in the comments, which can be found here, even when it's quite different, it was a lot of help.

http://coliru.stacked-crooked.com/a/ae64a70076436165

I was told the mallocs were the problems by and from irc. :) — Onza, Jun 22 '15 at 05:28
It would help a lot if you added comments to explain what you think the code is doing, or what it's supposed to do. Frankly, I find it baffling. What do you think you are allocating enough space for in the two `malloc` lines? — David Schwartz, Jun 22 '15 at 06:36

score 2 · Accepted Answer · answered Jun 22 '15 at 05:52

pub_l = malloc(sizeof(pub_l)); is simply not needed. Nor is priv_l = malloc(sizeof(priv_l));. Remove them both from your function.

You should be populating your out-parameters; instead you're throwing out the caller's provided addresses to populate and (a) populating your own, then (b) leaking the memory you just allocated.

The result is the caller's privl and publ are untouched and thus the decoding back to RSA is dysfunctional, as both values are indeterminate.

score 2 · Answer 2 · edited May 23 '17 at 12:06

pRSA = RSA_generate_key(RSA_LEN,RSA_FACTOR,NULL,NULL);

I think this is wrong. I know you are supposed to use RSA_generate_key_ex, and I think it needs a BIGNUM, not an integer. You should have gotten a warning. See RSA_generate_key(3) for details.

Your code should look something like:

BIGNUM* exp = BN_new();
ASSERT(exp != NULL);

int rc = BN_set_word(exp, RSA_F4);
ASSERT(rc == 1);

RSA* rsa = RSA_new();
ASSERT(rsa != NULL);

rc = RSA_generate_key_ex(rsa, 2048, exp, NULL);
ASSERT(rc == 1);

Be sure to call BN_free on the BIGNUM, and RSA_free on the RSA pointer.

RSA *privrsa = NULL;
d2i_RSAPrivateKey(&privrsa,(const unsigned char **)&priv,privl);

RSA *pubrsa = NULL;
d2i_RSAPublicKey(&pubrsa,(const unsigned char **)&pub,publ);

For this, it looks like you are trying to separate the public key and private key. For that, use RSAPublicKey_dup and RSAPrivateKey_dup. See Separating public and private keys from RSA keypair variable.

Its not clear to me what you are trying to do with the following. You should state what you are trying to do...

pub_l = malloc(sizeof(pub_l));
*pub_l = i2d_RSAPublicKey(pRSA,pub);
priv_l = malloc(sizeof(priv_l));
*priv_l = i2d_RSAPrivateKey(pRSA,priv);

I'm just guessing, but I'm going to say its all wrong. sizeof(priv_l) is the size of a pointer, so its 4 or 8 bytes. You're also overwriting the pointer passed in by the caller...

Also see OpenSSL's rsautl cannot load public key created with PEM_write_RSAPublicKey. It talks about saving the keys with SubjectPublicKeyInfo and PrivateKeyInfo in both ASN.1/DER and PEM formats.

By writing the {Public|Private}KeyInfo, the OID gets written to the key. That's important for interop. You also use the RSA* (and even an EVP_PKEY*), and not byte arrays.

No warning, the first part is ok even when I know I use old style (deprecated) code; the second answer is to get the ans1 representation so it's ok, and the deadly code was the pointers as mentioned. — Onza, Jun 23 '15 at 00:14

Segmentation fault with generating an RSA and saving in ASN.1/DER?

2 Answers2