C: OpenSSL RSA_private_decrypt() fails with "error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error”

Question

I'm new with cryptography, so I decided to create simple program that would open a file encrypt data, put it in etest.txt, then open this file decrypt it and put it indetest.txt.I know it sounds really weired but its for educational purposes. so here is my code. I've read many topics about this problem but none of them worked for me.

#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <stdio.h>
#include <string.h>

int main(void) {
    size_t pri_len;            // Length of private key
    size_t pub_len;            // Length of public key
    char   *pri_key;           // Private key
    char   *pub_key;           // Public key
    char   *msg = malloc(256);  // Message to encrypt
    char   *encrypt = NULL;    // Encrypted message
    char   *decrypt = NULL;    // Decrypted message
    char   *err;               // Buffer for any error messages
    size_t red;

    RSA *keypair = RSA_generate_key(2048, 3, NULL, NULL);
     FILE *in   = fopen("test.txt", "r");
     FILE *out  = fopen("etest.txt", "w");

     if(in == NULL)
     {  
        printf("in Error is %d (%s).\n", errno, strerror(errno));
     }
     if(out == NULL)
     {  
        printf("out Error is %d (%s).\n", errno, strerror(errno));
     }

    encrypt = malloc(RSA_size(keypair));

    for(;;)
    {
        red = fread(msg, 1, RSA_size(keypair)-42, in);
        if((RSA_public_encrypt(RSA_size(keypair)-42, (unsigned char*)msg, (unsigned char*)encrypt,
                                             keypair, RSA_PKCS1_OAEP_PADDING)) == -1) {
            ERR_load_crypto_strings();
            ERR_error_string(ERR_get_error(), err);
            fprintf(stderr, "Error encrypting message: %s\n", err);
        }

        if(fwrite(encrypt, 1, strlen(encrypt), out) == 1)
        {
           printf("fwrite Error is %d (%s).\n", errno, strerror(errno));
        }

        if(feof(in))
        {
            break;
        }       
    }

    fclose(in);
    fclose(out);

    in   = fopen("etest.txt", "r");
    out  = fopen("dtest.txt", "w");

     if(in == NULL)
     {        
        printf("in Error is %d (%s).\n", errno, strerror(errno));
     }

     if(out == NULL)
     {        
        printf("out Error is %d (%s).\n", errno, strerror(errno));
     }

    decrypt = malloc(RSA_size(keypair));

    for(;;)
    {
         red = fread(msg, 1, 256, in);
         if(RSA_private_decrypt(red, (unsigned char*)msg, (unsigned char*)decrypt,
                               keypair, RSA_PKCS1_OAEP_PADDING) == -1) {

            ERR_load_crypto_strings();
            ERR_error_string(ERR_get_error(), err);
            fprintf(stderr, "Error decrypting message: %s\n", err);
        }        

        fwrite(decrypt, 1, strlen(decrypt), out);

        if(feof(in))
        {
          break;
        }
    }
    fclose(in);
    fclose(out);     
    RSA_free(keypair);
    return 0;
}

When I run code it gives me back error saying:Error decrypting message: error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error Sorry if my question sound silly. Hope you can help. Thanks.

Answer 1

There are a few errors here. First when you're reading and encrypting:

red = fread(msg, 1, RSA_size(keypair)-42, in);
if((RSA_public_encrypt(RSA_size(keypair)-42, (unsigned char*)msg, (unsigned char*)encrypt,
                                     keypair, RSA_PKCS1_OAEP_PADDING)) == -1) {

A call to fread won't necessarily read the number of bytes asked for, and could return 0. So when you reach the end of the file, you man be encrypting more bytes than you need. So pass in red for the number of bytes to encrypt. Also, first check if red is 0 and if so break out of the loop:

red = fread(msg, 1, RSA_size(keypair)-42, in);
if (red == 0) break;
if(((red=RSA_public_encrypt(RSA_size(keypair)-42, (unsigned char*)msg, (unsigned char*)encrypt,
                                     keypair, RSA_PKCS1_OAEP_PADDING))) == -1) {

Note that we're saving the return value of RSA_public_encrypt. That comes into play here where you're writing the encrypted data to disk:

if(fwrite(encrypt, 1, strlen(encrypt), out) == 1)

encrypt is an array of characters, not a string. This means it's not NULL terminated, and it might contain NULL bytes. So you can't use strlen. Instead, capture the return value of RSA_public_encrypt and pass that as the size to write:

if(fwrite(encrypt, 1, red, out) == 1)

Because we're checking the return value of fread to break out of the loop, this isn't needed:

if(feof(in))
{
    break;
}

See this post regarding the perils of using feof.

Then there's this when you're reading back the encrypted data:

 red = fread(msg, 1, 256, in);
 if(RSA_private_decrypt(red, (unsigned char*)msg, (unsigned char*)decrypt,
                       keypair, RSA_PKCS1_OAEP_PADDING) == -1) {

RSA_private_decrypt expects a single encrypted block whose length is RSA_size(keypair). So read in that many bytes from disk and pass in that many bytes to the function. Also, check the return value of fread and break out if you don't get the expected amount, and capture the return value of RSA_private_decrypt:

 red = fread(msg, 1, RSA_size(keypair), in);
 if (red < RSA_size(keypair)) break;
 if((red=RSA_private_decrypt(red, (unsigned char*)msg, (unsigned char*)decrypt,
                       keypair, RSA_PKCS1_OAEP_PADDING)) == -1) {

Later when you write the decrypted data to disk:

fwrite(decrypt, 1, strlen(decrypt), out);

While what was decrypted is probably a string (if your input file was plain text), the returned data is not NULL terminated, so explicitly write than many bytes instead of using strlen:

fwrite(decrypt, 1, red, out);

Finally, as with the encryption loop, this is not needed in the decryption loop:

if(feof(in))
{
    break;
}

With these fixes applied, you should get the expected results.