Decrypt AES with wrong IV size

Question

I have a file encrypted in AES 128 CBC (with openssl in cpp). The encryption has been done with a wrong iv size (8 instead of 16). It's a fact, it's wrong and I can't do anything about it. The file I get is like this:
[8 bit of IV][Encrypted data]

I have to read this file with Java (android) but I cannot get a correct result.

Here is what I'm using to decrypt the file:

public String decrypt(byte[] key, byte[] datasencrypted) {
        // Split IV and actual Datas into 2  differents buffers
        int dataSize = datasencrypted.length - 8; // 8 = wrong iv size
        byte[] encrypted = new byte[dataSize];
        byte[] ivBuff = new byte[8];
        System.arraycopy(datasencrypted,0,ivBuff,0,8);
        System.arraycopy(datasencrypted,8,encrypted,0,dataSize);

        try {
            IvParameterSpec iv = new IvParameterSpec(ivBuff);
            SecretKeySpec skeySpec = new SecretKeySpec(key, "AES");

            Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
            cipher.init(Cipher.DECRYPT_MODE, skeySpec, iv);
            byte[] original = cipher.doFinal(encrypted);

            return new String(original);
        } catch (Exception ex) {
            ex.printStackTrace();
        }

        return null;
    }

Obviously this code throw me an InvalidAlgorithmParameterException: expected IV length of 16. If I change the size of the iv's buffer to 16 no more exception but the end result is just gibberish.

Also the key is defined as unsigned char[] in CPP so to convert it to byte[] in java I simply cast the values like this:

mKey =  new byte[]{(byte)0x8c,(byte)0x96,0x5f,.....}};

• Is it possible to decrypt this file in java (it's working on the cpp side)?
• How should I know which kind of padding to use?
• Does the cast of the key value may be a problem?

--- EDIT --- As suggested here is the CPP code that manage to decrypt the file :. AES_KEYLEN = 128.
AES_ROUND = 5.

size_t Helper::DecryptAES(unsigned char* encMsg, size_t encMsgLen, unsigned char** decMsg)
{
    size_t decLen = 0;
    size_t blockLen = 0;

    *decMsg = (unsigned char*)malloc(encMsgLen);
    if (*decMsg == nullptr) return 0;

    if (!EVP_DecryptUpdate(mAESDecryptCtx, (unsigned char*)*decMsg, (int*)&blockLen, encMsg, (int)encMsgLen)) {
        return 0;
    }
    decLen += blockLen;

    if (!EVP_DecryptFinal_ex(mAESDecryptCtx, (unsigned char*)*decMsg + decLen, (int*)&blockLen)) {
        return 0;
    }
    decLen += blockLen;

    return decLen;
}

bool Helper::InitAES(unsigned char* key, unsigned char* salt)
{
    mAESEncryptCtx = static_cast<EVP_CIPHER_CTX*>(malloc(sizeof(EVP_CIPHER_CTX)));
    mAESDecryptCtx = static_cast<EVP_CIPHER_CTX*>(malloc(sizeof(EVP_CIPHER_CTX)));

    mAESKey = static_cast<unsigned char*>(malloc(AES_KEYLEN / 8));
    mAESIv = static_cast<unsigned char*>(malloc(AES_KEYLEN / 8));

    int size = EVP_BytesToKey(EVP_aes_128_cbc(), EVP_sha256(), salt, key, AES_KEYLEN / 8, AES_ROUNDS, mAESKey, mAESIv);

    if (size != AES_KEYLEN / 8)
    {
        return false;
    }

    EVP_CIPHER_CTX_init(mAESEncryptCtx);
    if (!EVP_EncryptInit_ex(mAESEncryptCtx, EVP_aes_128_cbc(), nullptr, mAESKey, mAESIv))
        return false;

    EVP_CIPHER_CTX_init(mAESDecryptCtx);
    if (!EVP_DecryptInit_ex(mAESDecryptCtx, EVP_aes_128_cbc(), nullptr, mAESKey, mAESIv))
        return false;

    return true;
}

IvSIze is ok here , but it's when the iv is concatened to the file that only 8 bits are read instead of 16.

Answer 1

Both the key and IV are derived through EVP_BytesToKey, so I guess what is in front of the ciphertext is the salt and not the IV. This is however in code that you haven't shown.

If you want to implement the largely proprietary EVP_BytesToKey then there are multiple Java implementations available. Note that you will have to perform the split between key and IV yourself (16 bytes each).

Notes:

• CBC requires an IV of 16 bytes, it is impossible to perform CBC with an IV of 8 bytes (the implementation may try and fill in the missing bytes themselves or use data beyond the buffer in C of course);
• the C++ really makes a mess out of things, the IV size should e.g. not depend on the key size;
• while you're at it, please make a description of the protocol rather than to putting the next developer in the same position as you're in now;
• this is all using old tech, a bad key derivation mechanism, CBC encryption without authentication, I'd update your protocol.