Invalid characters generated while encrypting with openssl evp aes_256_ctr() mode

Question

My idea is to do file encryption in a client server model and i am using openssl evp for encryption purpose. I need to store the cipher text in a text file and send it to the client. But i am unable to do this because i find invalid characters being present in the cipher text which cannot be stored in a file.

This is my code for encryption :

EVP_CIPHER_CTX_init(&ctx);
EVP_CipherInit_ex(&ctx, EVP_aes_256_ctr(), NULL, NULL, NULL,
        do_encrypt);
OPENSSL_assert(EVP_CIPHER_CTX_key_length(&ctx) == 32);
OPENSSL_assert(EVP_CIPHER_CTX_iv_length(&ctx) == 16);

EVP_CipherInit_ex(&ctx, NULL, NULL, key, iv, do_encrypt);

//receive the file contents in chunks of 1024 bytes
while ((inlen = recv(connfd, inbuf, sizeof inbuf, 0)) > 0) {
    fprintf(stdout,"\nReceived %d bytes",inlen);
    fflush(stdout);
    fprintf(stdout,"\nOriginal: %s",inbuf);
    fflush(stdout);
    //use encrypt_update() to encrypt the chunks
    if(!EVP_CipherUpdate(&ctx, outbuf, &outlen, inbuf, inlen)) {
        /* Error */
        EVP_CIPHER_CTX_cleanup(&ctx);
        return 0;
    }
    //write the encrypted text to out file
    fprintf(stdout,"\nEncrypted: %s %d",outbuf, inlen);
    fflush(stdout);
    fwrite(outbuf, sizeof(char), outlen, fp);
    //clear the buffer
    memset(inbuf,0, strlen(inbuf));
    memset(outbuf,0, strlen(outbuf));
}
//use encrypt_final() to encrypt the final letf out block of chunk is any
if(!EVP_CipherFinal_ex(&ctx, outbuf, &outlen)) {
    /* Error */
    EVP_CIPHER_CTX_cleanup(&ctx);
    return 0;
}
//write the encrypted text to out file
fwrite(outbuf, sizeof(char), outlen, fp);
EVP_CIPHER_CTX_cleanup(&ctx); //cleanup
fclose(fp); //close the file

I referred this link where an issue of invalid characters with decryption being reported and solved.

Issues with encrypting a file using openssl evp api(aes256cbc)

I hope someone could help me out here.

Thanks in advance.

What makes you think the bytes emitted from an encryption are printable *characters* ? — WhozCraig, Dec 18 '14 at 07:07
i printed them just to check if something is being generated or not....i dont think they are printable....but i can't store them and transfer them also? — Sudershan, Dec 18 '14 at 07:10
Your error checking looks correct. Assuming a PKCS5 padding equivalence, check your output file to make sure it is multiple of the block size (16 bytes). If you want something displayable you can base64 or simple hex-encode the output bytes before sending them to the output file. of course, you'll also have to reverse that to get the original encrypted bytes back when performing a decryption. — WhozCraig, Dec 18 '14 at 07:13
yeah i will check the output file and yes i tried base64 but it doesn't seem to work right for me....i lost some data while decrypting.... — Sudershan, Dec 18 '14 at 07:16
Whilst not necessarily the cause of the issue you describe, the `memset`s you do at the end of each loop look suspicious - you're using `strlen` to determine the number of bytes to set to 0, yet these aren't null-terminated strings, so there's no guarantee that `strlen` will return the correct value and as a result you may be zeroing out unrelated memory. This could easily cause unexpected behavior or a crash under the right circumstances. — Iridium, Dec 18 '14 at 11:19
i used memset because if the received text in the current iteration is shorter than the one received in the previous iteration the remaining charters in the previous iteration which aren't overwritten stays in the buffer. i had to flush the old content. — Sudershan, Dec 18 '14 at 11:30

jww · Answer 1 · 2014-12-21T13:10:33.403

Invalid characters generated while encrypting with openssl evp aes_256_ctr() mode...

... because i find invalid characters being present in the cipher text which cannot be stored in a file

I think this is your problem. Its not quite correct.

You can store anything (any character) in a file. C-strings are a little different, but you are not working with a string.

All characters are equally probably in the cipher text (equally probable as any other character, like 0x00, 0x01, ... 'A', 'B', ... 'a', 'b', ..., 0xFE, 0xFF).

fprintf(stdout,"\nOriginal: %s",inbuf);

This could be a problem if inbuf has an embedded NULL. I thought you were dealing with files and not strings?

memset(inbuf,0, strlen(inbuf));
memset(outbuf,0, strlen(outbuf));

As Iridium said, these are not needed. You should be using the return values of functions like recv (and not depending on distinguished characters like NULL since its equally probably in the cipher text (equally probable as any other character, like 0x00, 0x01, ... 'A', 'B', ... 'a', 'b', ..., 0xFE, 0xFF).

EVP_CIPHER_CTX_init(&ctx);
EVP_CipherInit_ex(&ctx, EVP_aes_256_ctr(), NULL, NULL, NULL, do_encrypt);
...

Your also ignoring return values. That's usually a bad idea.

Invalid characters generated while encrypting with openssl evp aes_256_ctr() mode

1 Answers1