OpenSSL in C: after second decryption in application run, the first 16 bytes of result are garbage

Question

I implemented simple file encryption/decryption with OpenSSL in C according to the instructions here. I do not need this to be truly secure (just want the files to not be readily readable on drive), the keys are hardcoded in the application and after reading the encrypted files from drive I decrypt them.

On first call, the decryptFileAsBytes function returns the correct decrypted file as byte vector. On the second call (within the same application run) the first 16 bytes of the result are garbage and the rest is correct. Does this have something to do with the size of the key (128 bits) I am using?

static bool decryptFileAsBytes(std::string filename, unsigned char *ckey, unsigned char *ivec, std::vector<unsigned char> &fileBytes)
{
    std::ifstream ifs(filename, std::ios::binary | std::ios::ate);
    if (ifs.fail())
        return false;

    std::ifstream::pos_type pos = ifs.tellg();
    fileBytes.resize(pos);
    ifs.close();

    FILE *ifp;
    if (fopen_s(&ifp, filename.c_str(), "rb") != NULL)
        return false;

    int bytesRead;
    unsigned char indata[AES_BLOCK_SIZE];
    unsigned char *writePtr = fileBytes.data();

    /* data structure that contains the key itself */
    AES_KEY key;

    /* set the encryption key */
    AES_set_encrypt_key(ckey, 128, &key);

    /* set where on the 128 bit encrypted block to begin encryption*/
    int num = 0;

    while (1)
    {
        bytesRead = fread(indata, 1, AES_BLOCK_SIZE, ifp);

        AES_cfb128_encrypt(indata, writePtr, bytesRead, &key, ivec, &num, AES_DECRYPT);
        writePtr += bytesRead;

        if (bytesRead < AES_BLOCK_SIZE)
            break;
    }
    if (fclose(ifp) != NULL)
        return false;

    return true;
}

Alternatively to solving this, I welcome suggestions of a simple solution to the problem stated above ('encrypt' file on drive in a not bulletproof way so that it is not readily readable but the application can decrypt it).

Curious what you discovered while debugging this, specifically the content of the memory behind your `ivec` when you embark upon your first, then subsequent decryptions. — WhozCraig, Sep 22 '18 at 14:21
@WhozCraig You were right on target. I assumed that the memory pointed to by `ivec` does not change but it did, and was garbage after the first encryption. After creating a copy and passing that into `AES_cfb128_encrypt` it works as expected. Thanks! Feel free to post this as an answer. — pseudomarvin, Sep 22 '18 at 15:42

score 1 · Accepted Answer · answered Sep 22 '18 at 15:51

The problem is likely that you're not retaining the original initialization vector for subsequent decryption operations.

As the AES encryption/decryption operations transpire, that memory is updated to continue with subsequent frames. If you instrument your code you'll see, with each encrypt/decrypt frame passing through the API, the ivec is changed.

If all you're doing this for is obfuscation (eg. you have a static key in your application) my suggestion is to do the following:

Don't pass the ivec into either the encryptor or decryptor.
Instead, generate a random ivec using RAND_bytes when encrypting. Store the ivec as the first block of data before continuing with the file content.
When decrypting, read the first block of data to prime your ivec.
Then, decrypt the remainder of the file as normal.

The benefits are:

Each encryption of a file will create a different byte representation, dependent on the initial random ivec. Eg. if you encrypt a file twice the resulting encrypted bytes will not be the same
You no longer have to use a static ivec from somewhere else in your code. The file contains it as the first block of data.

Just a suggestion. Unrelated, I prefer the EVP encryption interface, and suggest it worth a look.

OpenSSL in C: after second decryption in application run, the first 16 bytes of result are garbage

1 Answers1