d2i_RSA_PUBKEY, d2i_RSAPrivateKey and d2i_RSAPublicKey returns NULL

Question

I created a RSA private key using the following command:

 openssl genrsa -out keypair.pem 2048

I must to use DER-encoded keys (PKCS#1) for this project, so I generated two DER files from this PEM-encoded private key file - one with the private key and another with the public key.

openssl rsa -inform PEM -in keypair.pem -outform DER -pubout -out public.der

openssl rsa -inform PEM -in keypair.pem -outform DER -out private.der

In my code, I loaded the contents of both these files in char* variables.

None of the following works as expected:

d2i_RSA_PUBKEY(NULL, &public_key_bytes, public_key_length);

d2i_RSAPublicKey(NULL, &public_key_bytes, public_key_length);

d2i_RSAPrivateKey(NULL, &private_key_bytes, private_key_length);

I know that beacause all of them return null. I also tried the following:

RSA * rsa = RSA_new();
d2i_RSA_PUBKEY(&rsa, &public_key_bytes, public_key_length);

RSA * rsa = RSA_new();
d2i_RSAPublicKey(&rsa, &public_key_bytes, public_key_length);

RSA * rsa = RSA_new();
d2i_RSAPrivateKey(&rsa, &private_key_bytes, private_key_length);

All return null as well.

My complete test code is as follows:

#include <stdio.h>
#include <stdlib.h>

#include <openssl/rsa.h>
#include <openssl/bio.h>
#include <openssl/pem.h>

typedef struct
{
    int len;
    char * bytes;
} FileData;

static FileData readFileBytes(const char * name, int zero_ended)
{
    FILE * fl = fopen(name, "r");
    if (fl == NULL) return (FileData) { .len = 0, .bytes = NULL };
    fseek(fl, 0, SEEK_END);
    long len = ftell(fl);
    char * ret = malloc(len + (zero_ended ? 1 : 0));
    fseek(fl, 0, SEEK_SET);
    fread(ret, 1, len, fl);
    if (zero_ended) ret[len] = 0;
    fclose(fl);
    return (FileData) { .len = len, .bytes = ret };
}

int main()
{
    FileData private_key = readFileBytes("../private.der", 0);
    FileData public_key = readFileBytes("../public.der", 0);

    char* public_key_bytes = public_key.bytes;
    int public_key_length = public_key.len;

    char* private_key_bytes = private_key.bytes;
    int private_key_length = private_key.len;

    RSA * rsa;

    public_key_bytes = public_key.bytes;
    public_key_length = public_key.len;
    rsa = d2i_RSA_PUBKEY(NULL, &public_key_bytes, public_key_length);
    printf("d2i_RSA_PUBKEY(NULL, &public_key_bytes, public_key_length) != NULL -> %s\n", (rsa != NULL) ? "true" : "false");

    public_key_bytes = public_key.bytes;
    public_key_length = public_key.len;
    rsa = d2i_RSAPublicKey(NULL, &public_key_bytes, public_key_length);
    printf("d2i_RSAPublicKey(NULL, &public_key_bytes, public_key_length) != NULL -> %s\n", (rsa != NULL) ? "true" : "false");

    private_key_bytes = private_key.bytes;
    private_key_length = private_key.len;
    rsa = d2i_RSAPrivateKey(NULL, &private_key_bytes, private_key_length);
    printf("d2i_RSAPrivateKey(NULL, &private_key_bytes, private_key_length) != NULL -> %s\n", (rsa != NULL) ? "true" : "false");

    public_key_bytes = public_key.bytes;
    public_key_length = public_key.len;
    rsa = RSA_new();
    rsa = d2i_RSA_PUBKEY(&rsa, &public_key_bytes, public_key_length);
    printf("d2i_RSA_PUBKEY(&rsa, &public_key_bytes, public_key_length) != NULL -> %s\n", (rsa != NULL) ? "true" : "false");

    public_key_bytes = public_key.bytes;
    public_key_length = public_key.len;
    rsa = RSA_new();
    rsa = d2i_RSAPublicKey(&rsa, &public_key_bytes, public_key_length);
    printf("d2i_RSAPublicKey(&rsa, &public_key_bytes, public_key_length) != NULL -> %s\n", (rsa != NULL) ? "true" : "false");

    private_key_bytes = private_key.bytes;
    private_key_length = private_key.len;
    rsa = RSA_new();
    rsa = d2i_RSAPrivateKey(&rsa, &private_key_bytes, private_key_length);
    printf("d2i_RSAPrivateKey(&rsa, &private_key_bytes, private_key_length) != NULL -> %s\n", (rsa != NULL) ? "true" : "false");

    getchar();

    return 0;
}

What am I doing wrong?

Answer 1

TLDR: d2i_RSA_PUBKEY and d2i_RSAPrivateKey should work, and do for me on Unix.

You say you want 'DER-encoded keys (PKCS#1)'.

For publickeys OpenSSL normally uses the format defined in X.509 as SubjectPublicKeyInfo which contains an AlgorithmIdentifier plus (edited) BIT STRING containing a publickey value in a structure that varies depending on the algorithm. For RSA, the algid contains an OID identifying RSA and no parameters; the algorithm-dependent structure is PKCS#1.

In contrast, OpenSSL supports two types of privatekey format: there is a 'legacy' format for each algorithm (except DH), which for RSA is PKCS#1; and a generic format defined by PKCS#8 which like SPKI consists of an AlgorithmIdentifier plus an algorithm-dependent privatekey value, this time in OCTET STRING. PKCS#8 also has an option to encrypt the key, which SPKI doesn't have or need.

Older parts of OpenSSL, including the genrsa and rsa commandline utilities, use legacy privatekey format, but SPKI publickey format which OpenSSL names PUBKEY. Thus your rsa commands created a publickey file readable by d2i_RSA_PUBKEY but not d2i_RSAPublicKey (which would be only the PKCS#1 part) and a privatekey file readable by d2i_RSAPrivateKey.

If you truly need publickey in 'bare' PKCS#1 format, the rsa utility has options -RSAPublicKey_in and -RSAPublicKey_out to read and write this format since 1.0.0, although documented only recently and still not in the help message. That file will be readable by d2i_RSAPublicKey but not d2i_RSA_PUBKEY.

One possibility: you don't mention operating system. DER files are binary, and in C to correctly handle binary files on Windows you must fopen with the b modifier, here you want "rb" for read binary. If I run your code on Unix it works, but to get correct results on Windows I must add the b.

Also a minor point: you talk about 'loading contents ... in char* variables'. Actually you load the file contents into memory and use a char * variable to point to them. Strictly speaking, OpenSSL d2i routines want the address of a const unsigned char * variable -- and your compiler should warn you about this mismatch at least if you run it in a standard-conforming mode. But C requires pointers to all char flavors (signed unsigned and 'plain'), with or without qualification, to have the same representation and alignment requirements, even though they aren't compatible as defined in the standard, so passing char ** where const unsigned char ** is expected does work.