Function is not generating proper openssl rsa keys

Question

This is a c function I wrote to generate openssl rsa 4096 bit keys.

    bool rsa_gen_keys()
    {    
    int             ret = 0;
    RSA             *rsa = NULL;
    BIGNUM          *bignum = NULL;
    BIO             *bio_private = NULL;
    BIO             *bio_public = NULL;
    int              bits = 4096;

    unsigned long k = RSA_F4;

    bignum = BN_new();
    ret = BN_set_word(bignum,k);
    if(ret != 1){
        goto cleanup;
    }

    rsa = RSA_new();
    ret = RSA_generate_key_ex(rsa, bits, bignum, NULL);
    if(ret != 1){
        goto cleanup;
    }
    // write rsa private key to file
    bio_private = BIO_new_file("private_new.pem", "w+");
    ret = PEM_write_bio_RSAPrivateKey(bio_private, rsa, NULL, NULL, 0, NULL, NULL);
    BIO_flush(bio_private);
    // write rsa public key to file
    bio_public = BIO_new_file("public_new.pem", "w+");
    ret = PEM_write_bio_RSAPublicKey(bio_public, rsa);
    if(ret != 1){
        goto cleanup;
    }    
    BIO_flush(bio_public);
cleanup:
    BIO_free_all(bio_private);
    BIO_free_all(bio_public);
    RSA_free(rsa);
    BN_free(bignum);
    return ret;
}

The keys generated by the above function seem to be missing something. When I try to use the public_new.pem file in another program, I get the following error:

140286309791384:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: PUBLIC KEY

However, if I use the openssl command to generate the key files, the files work fine.

$openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:4096

I noticed the key sizes generated from the function and from the command line don't match. This is a clue, but what do I need to change in my function to fix this?

-rw-rw-r-- 1 3272 Feb  6 09:19 private_key.pem
-rw-rw-r-- 1  800 Feb  6 09:20 public_key.pem
-rw-rw-r-- 1 3243 Feb  6 10:43 private_new.pem
-rw-rw-r-- 1  775 Feb  6 10:43 public_new.pem

BTW, I tried the above with 2048 bits keys and I get the same result and same size mismatch

What form or encoding do you want? You can see the different encodings at [Use OpenSSL RSA key with .Net](http://stackoverflow.com/q/30475758/608639). — jww, Feb 06 '17 at 16:41

score 2 · Answer 1 · answered Feb 06 '17 at 16:30

2

openssl genpkey is using PEM_write_bio_PrivateKey (PKCS#8) instead of PEM_write_bio_RSAPrivateKey (PKCS#1): https://github.com/openssl/openssl/blob/master/apps/genpkey.c#L161-L164.

You don't show how you generated public_key.pem, but it was probably written with PEM_write_bio_PUBKEY (X.509 SubjectPublicKeyInfo) vs PEM_write_bio_RSAPublicKey (PKCS#1).

From a PEM armor perspective:

PKCS#1 Public: BEGIN RSA PUBLIC KEY
X.509 SubjectPublicKeyInfo: BEGIN PUBLIC KEY
PKCS#1 Private: BEGIN RSA PRIVATE KEY
PKCS#8: BEGIN PRIVATE KEY

answered Feb 06 '17 at 16:30

bartonjs

30,352
2
71
111

That explains the difference. I may need to switch to using EVP calls to generate the correct format instead of direct RSA call .. like EVP_PKEY_keygen()?. – seedhom Feb 06 '17 at 17:00
1

Yep. The `EVP_*` (envelope(d)) structure, and the relevant file formats are about "what kind of key is this" and "oh, here's the key". Using the `RSA` direct APIs means you should already know what key type it is, so it leaves that out. – bartonjs Feb 06 '17 at 17:04

score 1 · Answer 2 · answered Feb 06 '17 at 18:42

I realized I needed to use the format of the keys for PKCS#8 and X.509. So I switched to EVP functions to generate them. Here is a very simplified version of the code I ended up using (no error checking):

bool rsa_gen_keys() {
    int ret = 0;
    BIO *bio_private = NULL;
    BIO *bio_public = NULL;
    int bits = 4096;

    EVP_PKEY_CTX *ctx;
    EVP_PKEY *pkey = NULL;

    // Get the context
    ctx = EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL);
    if (!ctx)
        goto cleanup;

    // init keygen
    if (EVP_PKEY_keygen_init(ctx) <= 0)
        goto cleanup;

    // set the bit size 
    if (EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, bits) <= 0)
    goto cleanup;

    /* Generate key */
    if (EVP_PKEY_keygen(ctx, &pkey) <= 0)
        goto cleanup;


    // write rsa private key to file
    bio_private = BIO_new_file("private_new.pem", "w+");
    ret = PEM_write_bio_PrivateKey(bio_private, pkey, NULL, NULL, 0, NULL, NULL);
    if (ret != 1) {
        goto cleanup;
    }
    BIO_flush(bio_private);

    // write rsa public key to file
    bio_public = BIO_new_file("public_new.pem", "w+");

    //ret = PEM_write_bio_RSAPublicKey(bio_public, rsa);
    ret = PEM_write_bio_PUBKEY(bio_public, pkey);
    if (ret != 1) {
        goto cleanup;
    }
    BIO_flush(bio_public);


cleanup:
    if(bio_private) BIO_free_all(bio_private);
    if(bio_public) BIO_free_all(bio_public);
    if(pkey) EVP_PKEY_free(pkey);

    return ret;
}

Function is not generating proper openssl rsa keys

2 Answers2