Decrypting a file with AES_256_CBC return "bad decrypt" error

Question

This is a follow up question from this one:

OpenSSL EVP_DecryptFinal_ex returns "wrong final block length" error when decrypting a file

I am trying to decrypt a file. At first I was reading it as ASCII file instead of binary. Having this fixed (I hope) and reading it as a binary I always get the an "bad decrypt" error:

15208:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto\evp\evp_enc.c:570:

Here is a sample for how I am encrypting and decrypting:

Encryption:

Cipher cipher;
ifstream f("d:/test.YML");
ofstream out("d:/temp.YML");
byte key[KEY_SIZE] = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2};
byte iv[BLOCK_SIZE] = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6};

secure_string line;
secure_string temp;
while (getline(f, line)) {
    cipher.Encrypt(key, iv, line, temp);
    out << temp << endl;
}

Decryption:

Cipher cipher;
    ifstream f("d:/temp.YML", ifstream::binary);
    ofstream out("d:/tempDecrypt.YML");
    byte key[KEY_SIZE] = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2};
    byte iv[BLOCK_SIZE] = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6};

    secure_string temp;
    vector<char> buffer(1024, 0);

    while (!f.eof()) {
        f.read(buffer.data(), buffer.size());
        streamsize dataSize = f.gcount();
        secure_string chunk = { buffer.begin(), buffer.begin() + dataSize };

        cipher.Decrypt(key, iv, chunk, temp);
    }

Now I am not sure where start with this investigation:

1. Is there an encryption problem ? The encrypted file is generated, I am not seeing anything wrong with it.

2. Is there a problem with how I am reading chunks of file and decrypt them ? Again I don't see the issue here.(the error is on EVP_DecryptFinal_ex)

3. I also heard there could be a problem with padding. I am not doing anything related to padding, so I am not sure if this is an issue.

4. I am using same version of OpenSsl, on Windows, I have 2 Visual Studio projects, so there shouldn,t be an issue with imcompatible OpenSsl libraries.

If someone has any pointers pleaase let me know. I never worked with encryption before so some things are hard to understand.

PS: I haven't included the Encrypt and Decrypt methods, they are the same as on the Openssl Wiki website, let me know if I should.

Answer 1

There are a number of issues in your code... to name a few:

• ofstream out("d:/temp.YML"); should be opened in binary mode.
• out << temp << endl; will damage binary (encrypted) data by adding newlines unnecessarily.
• Output buffer should contain enough space to fit (input buffer + block_size).
• Encryption/decryption in chunks must follow the update/final pattern. You can't encrypt/decrypt chunks independently.
• The IV should be random and should be stored with the ciphertext.

Have a look at the following sample application, which works:

#include <cstdint>
#include <fstream>
#include <openssl/conf.h>
#include <openssl/evp.h>
#include <openssl/err.h>
#include <openssl/rand.h>
#include <string.h>
#include <stdio.h>

static const size_t KEY_SIZE = 256 / 8, BLOCK_SIZE = 128 / 8;

class AESBase {
protected:
    const uint8_t *key, *iv;
    EVP_CIPHER_CTX *ctx;
    AESBase(const uint8_t *key, const uint8_t *iv) : key(key), iv(iv) {
        if (!(ctx = EVP_CIPHER_CTX_new()))
            handleErrors();
    }
    ~AESBase() {
        EVP_CIPHER_CTX_free(ctx);
    }
    static void handleErrors(void) {
        ERR_print_errors_fp(stderr);
        abort();
    }
};

class Encrypt : AESBase {
public:
    Encrypt(const uint8_t *key, const uint8_t *iv) : AESBase(key, iv) {
        if (1 != EVP_EncryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv))
            handleErrors();
    }
    int update(const char *plaintext, int plaintext_len, char *ciphertext) {
        int len;
        if (1 != EVP_EncryptUpdate(ctx, (uint8_t*)ciphertext, &len, (const uint8_t*)plaintext, plaintext_len))
            handleErrors();
        return len;
    }
    int final(char *ciphertext) {
        int len;
        if (1 != EVP_EncryptFinal_ex(ctx, (uint8_t*)ciphertext, &len))
            handleErrors();
        return len;
    }
};

class Decrypt : AESBase {
public:
    Decrypt(const uint8_t *key, const uint8_t *iv) : AESBase(key, iv) {
        if (1 != EVP_DecryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv))
            handleErrors();
    }
    int update(const char *ciphertext, int ciphertext_len, char *plaintext) {
        int len;
        if (1 != EVP_DecryptUpdate(ctx, (uint8_t*)plaintext, &len, (const uint8_t*)ciphertext, ciphertext_len))
            handleErrors();
        return len;
    }
    int final(char *plaintext) {
        int len;
        if (1 != EVP_DecryptFinal_ex(ctx, (uint8_t*)plaintext, &len))
            handleErrors();
        return len;
    }
};

void test_encrypt(const uint8_t *key, const char* in, const char* out) {
    std::ifstream fin(in, std::ios_base::binary);
    std::ofstream fout(out, std::ios_base::binary);
    uint8_t iv[BLOCK_SIZE];
    RAND_bytes(iv, sizeof(iv));

    char buf[1024], temp[sizeof(buf) + BLOCK_SIZE];
    Encrypt aes(key, iv);
    fout.write((char*)iv, sizeof(iv));
    while (fin) {
        fin.read(buf, sizeof(buf));
        int len = (int)fin.gcount();
        if (len <= 0)
            break;
        len = aes.update(buf, len, temp);
        fout.write(temp, len);
    }
    int len = aes.final(temp);
    fout.write(temp, len);
}

void test_decrypt(const uint8_t *key, const char* in, const char* out) {
    std::ifstream fin(in, std::ios_base::binary);
    std::ofstream fout(out, std::ios_base::binary);
    uint8_t iv[BLOCK_SIZE];
    fin.read((char*)iv, sizeof(iv));

    char buf[1024], temp[sizeof(buf) + BLOCK_SIZE];
    Decrypt aes(key, iv);
    while (fin) {
        fin.read(buf, sizeof(buf));
        int len = (int)fin.gcount();
        if (len <= 0)
            break;
        len = aes.update(buf, len, temp);
        fout.write(temp, len);
    }
    int len = aes.final(temp);
    fout.write(temp, len);
}

int main()
{
    ERR_load_crypto_strings();
    OpenSSL_add_all_algorithms();
    OPENSSL_config(NULL);

    uint8_t key[KEY_SIZE] = { 1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2 };
    test_encrypt(key, "main.cpp", "main.cpp.enc");
    test_decrypt(key, "main.cpp.enc", "main.cpp.txt");
}