Segfault in AES_cbc_encrypt

Question

I'm trying to understand the OpenSSL library in more detail. So rather than using the set of higher-level EVP functions, I've been trying to use the AES_* functions. Following the general set of calls in this question (though I'm using CBC instead of counter mode), I've come up with this code:

void ctr(log_t* log)
{
   unsigned char ivec[16];
   /* Out buffer for ciphertext */
   unsigned char outBuf[16];

   blockReader_t* br = blockReaderInit(log, "./input.txt", 128);
   int outFD;

   if ((outFD = open("out.bin", O_WRONLY)) == -1)
   {
      logPrint(br->log, LOG_ARGS, LOG_ERR, "open: %s", strerror(errno));
      logExit(br->log, LOG_ARGS, EXIT_FAILURE);
   }

   memset(ivec, 0, 16);

   unsigned char* ivec2 = ivec + 8;
   unsigned long* ivec3 = (unsigned long*) ivec2;
   *ivec3 = (unsigned long) 0xfd0;

   AES_KEY aesKey;
   char* myKey = "Pampers baby-dry";
   int res;

   if (!(res = AES_set_encrypt_key((unsigned char*) myKey, 16, &aesKey)))
   {
      logPrint(log, LOG_ARGS, LOG_ERR, "AES_set_encrypt_key: returned %d", res);
      logExit(log, LOG_ARGS, EXIT_FAILURE);
   }

   unsigned char* buf;

   while ((buf = blockReaderGet(br)) != NULL)
   {
      logPrint(log, LOG_ARGS, LOG_INFO, "ivec =");
      logHexdump(log, LOG_ARGS, LOG_INFO, (char*) ivec, 16);

      logPrint(log, LOG_ARGS, LOG_INFO, "buf =");
      logHexdump(log, LOG_ARGS, LOG_INFO, (char*) buf, 16);

      AES_cbc_encrypt(buf, outBuf, 16, &aesKey, ivec, 1);

      logPrint(log, LOG_ARGS, LOG_INFO, "outBuf =");
      logHexdump(log, LOG_ARGS, LOG_INFO, (char*) outBuf, 16);

      int res = write(outFD, outBuf, 16);

      if (res == -1)
      {
         logPrint(log, LOG_ARGS, LOG_ERR, "write: %s", strerror(errno));
         logExit(log, LOG_ARGS, EXIT_FAILURE);
      }
      else if (res < 16)
      {
         logPrint(log, LOG_ARGS, LOG_WARN, "Unexpectedly wrote < 16 bytes");
      }
   }

   if ((close(outFD)) == -1)
   {
      logPrint(log, LOG_ARGS, LOG_ERR, "close: %s", strerror(errno));
      logExit(log, LOG_ARGS, EXIT_FAILURE);
   }
}

The log_t struct and calls to log*() are my own logging framework which I am using to help debug this code. blockReader_t is another framework for reading files in sets of bytes. blockReaderGet() simply fills the destination buffer with the predetermined number of bytes of data (in this case 128 bits/16 bytes).

Contents of input.txt:

$ hexdump -C input.txt
00000000  4d 69 64 6e 69 67 68 74  5f 4d 61 72 6c 69 6e 05  |Midnight_Marlin.|
00000010  52 69 63 68 61 72 64 52  69 63 68 61 72 64 06 07  |RichardRichard..|
00000020

Output (ran in GDB):

(gdb) run
Starting program: /home/adam/crypto/openssl/aes/aes_128
[    0.000020] <aes_128.c:83> "main" INFO: Log library started (v1.9.0)
...
[    0.000054] <aes_128.c:50> "ctr" INFO: ivec =
[    0.000057] <aes_128.c:51> "ctr" INFO: HEX (16 bytes)
---BEGIN_HEX---
00000000  00 00 00 00 00 00 00 00  d0 0f 00 00 00 00 00 00  |................|
00000010
---END_HEX---
[    0.000069] <aes_128.c:53> "ctr" INFO: buf =
[    0.000071] <aes_128.c:54> "ctr" INFO: HEX (16 bytes)
---BEGIN_HEX---
00000000  4d 69 64 6e 69 67 68 74  5f 4d 61 72 6c 69 6e 05  |Midnight_Marlin.|
00000010
---END_HEX---

Program received signal SIGSEGV, Segmentation fault.
_x86_64_AES_encrypt_compact () at aes-x86_64.s:170
170             xorl    0(%r15),%eax

I'm using an OpenSSL from GitHub that I've built myself and linked against locally; specifically the OpenSSL_1_0_2e tag, which I gather is the latest stable version.

The Perl file that generates this assembly file uses the $key variable to name what r15 represents. But given that AES_set_encrypt_key() returns success, I'm not sure what's wrong.

Could anyone please offer any pointers to what might be wrong here?

EDIT:

Despite compiling OpenSSL with -g3 instead of -O3, the backtrace isn't useful:

(gdb) bt
#0  _x86_64_AES_encrypt_compact () at aes-x86_64.s:170
#1  0x0000000000402b6b in AES_cbc_encrypt () at aes-x86_64.s:1614
#2  0x00007fffffffe0a0 in ?? ()
#3  0x000080007dfc19a0 in ?? ()
#4  0x00007fffffffe050 in ?? ()
#5  0x0000000000635080 in ?? ()
#6  0x00007fffffffe1a0 in ?? ()
#7  0x0000000000000010 in ?? ()
#8  0x00007ffff7bdf9a0 in ?? ()
#9  0x00007fffffffe1b0 in ?? ()
#10 0x00007fff00000001 in ?? ()
#11 0x00007ffff7bdf4c8 in ?? ()
#12 0x00007fffffffda40 in ?? ()
#13 0x0000000000000000 in ?? ()

EDIT 2:

CFLAG has been changed:

CFLAG= -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -DL_ENDIAN -O0 -ggdb -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM

Note the -O0 -ggdb. Backtrace is the same:

(gdb) bt
#0  _x86_64_AES_encrypt_compact () at aes-x86_64.s:170
#1  0x0000000000402b6b in AES_cbc_encrypt () at aes-x86_64.s:1614
#2  0x00007fffffffe0a0 in ?? ()
#3  0x000080007dfc19a0 in ?? ()
#4  0x00007fffffffe050 in ?? ()
#5  0x0000000000635080 in ?? ()
#6  0x00007fffffffe1a0 in ?? ()
#7  0x0000000000000010 in ?? ()
#8  0x00007ffff7bdf9a0 in ?? ()
#9  0x00007fffffffe1b0 in ?? ()
#10 0x00007fff00000001 in ?? ()
#11 0x00007ffff7bdf4c8 in ?? ()
#12 0x00007fffffffda40 in ?? ()
#13 0x0000000000000000 in ?? ()

EDIT: MCVE example

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <openssl/aes.h>

unsigned char input[] = {0x4du, 0x69u, 0x64u, 0x6eu, 0x69u, 0x67u, 0x68u, 0x74u,
                         0x5fu, 0x4du, 0x61u, 0x72u, 0x6cu, 0x69u, 0x6eu, 0x05u,
                         0x52u, 0x69u, 0x63u, 0x68u, 0x61u, 0x72u, 0x64u, 0x52u,
                         0x69u, 0x63u, 0x68u, 0x61u, 0x72u, 0x64u, 0x06u, 0x07u};

int main()
{
   unsigned char ivec[16];
   /* ivec[0..7] is the IV, ivec[8..15] is the big endian counter. */
   unsigned char outBuf[16];

   int outFD;

   if ((outFD = open("out.bin", O_WRONLY)) == -1)
   {
      perror("open");
      return EXIT_FAILURE;
   }

   memset(ivec, 0, 16);

   unsigned char* ivec2 = ivec + 8;
   unsigned long* ivec3 = (unsigned long*) ivec2;
   *ivec3 = (unsigned long) 0xfd0;

   AES_KEY aesKey;
   char* myKey = "Pampers baby-dry";
   int res;

   if (!(res = AES_set_encrypt_key((unsigned char*) myKey, 16, &aesKey)))
   {
      fprintf(stderr, "AES_set_encrypt_key: returned %d", res);
      return EXIT_FAILURE;
   }

   for (int i = 0; i < 32; i += 16)
   {
      printf("ivec = ");

      for (int j = 0; j < 16; j++)
         printf("%.02hhx ", ivec[j]);

      putchar('\n');

      printf("input = ");

      for (int j = i; j < (i + 16); j++)
         printf("%.02hhx ", input[j]);

      putchar('\n');

      AES_cbc_encrypt(&input[i], outBuf, 16, &aesKey, ivec, 1);

      printf("outBuf = ");

      for (int j = 0; j < 16; j++)
         printf("%.02hhx ", outBuf[j]);

      putchar('\n');

      int res = write(outFD, outBuf, 16);

      if (res == -1)
      {
         perror("write");
         return EXIT_FAILURE;
      }
      else if (res < 16)
      {
         printf("Warning: unexpectedly wrote < 16 bytes");
      }
   }

   if ((close(outFD)) == -1)
   {
      perror("close");
      return EXIT_FAILURE;
   }

   return EXIT_SUCCESS;
}

What happens when you issue the `bt` (back/stack trace) command in GDB after the crash occurs? — Cloud, Jan 07 '16 at 23:05
Try compiling your app and OpenSSL via `-O0 -ggdb` to disable optimizations, enable debugging, etc. — Cloud, Jan 07 '16 at 23:37
@Dogbert Done, though it doesn't seem to have made any difference. — Doddy, Jan 07 '16 at 23:57
Hmm, without any more stack dump info, we're sort of flying blind unless someone who knows the code better can comment. Are you able to post a complete minimal working example here or on pastebin? [MCVW](http://stackoverflow.com/help/mcve). I could build and test it locally. — Cloud, Jan 08 '16 at 00:16
You should *not* use `AES_encrypt` and friends. That's a software-only implementation, so you will not enjoy hardware support, like AES-NI. You should be using `EVP_*` functions. See [EVP Symmetric Encryption and Decryption](https://wiki.openssl.org/index.php/EVP_Symmetric_Encryption_and_Decryption) on the OpenSSL wiki. In fact, you should probably be using authenticated encryption because it provides *both* confidentiality and authenticity. See [EVP Authenticated Encryption and Decryption](https://wiki.openssl.org/index.php/EVP_Authenticated_Encryption_and_Decryption) on the OpenSSL wiki. — jww, Jan 09 '16 at 16:17

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

So there are several major bugs here. I'll go through all the ones I caught, but there may be more, as I didn't do a thorough code review.

You are using sentinel values everywhere (ie: the 16 integer literals. Swap these out with a preprocessor macro, or even better, a const int).
The output buffer needs to be at least as big as your input buffer, and should be rounded up the the nearest multiple of the block size, plus one more block.
You are looping through each element of the the input data and trying to encrypt one byte at a time. Unless you are implementing some obscure layer on top of AES, this is wrong. You iterate over blocks of data, not individual bytes. The loop is completely unnecessary.
Your input data buffer appears to be bigger than your output data buffer. With your current implementation, the last 16 bytes I think will be truncated/lost, since the input buffer has 32 bytes of data, but the output buffer is 16 bytes. In your specific example, input should be 32 bytes, output should be 32+1.
In addition to the loop being unnecessary, with some modifications it would run (incorrectly, corrupting data), and eventually access invalid memory (ie: pointing to near the end of the input buffer, and telling the encrypt function to ask for 16 bytes of data after that point).

I've provided an updated code listing and sample output that should get you on the right track. Here's a working example that should also guide you along.

Good luck!

Modified Code Listing

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <openssl/aes.h>

#define BLOCK_SIZE  (128)

unsigned char input[BLOCK_SIZE] = {
    0x4du, 0x69u, 0x64u, 0x6eu, 0x69u, 0x67u, 0x68u, 0x74u,
    0x5fu, 0x4du, 0x61u, 0x72u, 0x6cu, 0x69u, 0x6eu, 0x05u,
    0x52u, 0x69u, 0x63u, 0x68u, 0x61u, 0x72u, 0x64u, 0x52u,
    0x69u, 0x63u, 0x68u, 0x61u, 0x72u, 0x64u, 0x06u, 0x07u};

int main()
{
    unsigned char ivec[BLOCK_SIZE];
    /* ivec[0..7] is the IV, ivec[8..15] is the big endian counter. */
    unsigned char outBuf[BLOCK_SIZE+1];

    int outFD;

    if ((outFD = open("out.bin", O_CREAT | O_RDWR)) == -1)
    {
        perror("open");
        return EXIT_FAILURE;
    }

    memset(ivec, 0, BLOCK_SIZE);

    unsigned char* ivec2 = ivec + 8;
    unsigned long* ivec3 = (unsigned long*) ivec2;
    *ivec3 = (unsigned long) 0xfd0;

    AES_KEY aesKey;
    char* myKey = "Pampers baby-dry";
    int res;

    if ((res = AES_set_encrypt_key((unsigned char*) myKey, BLOCK_SIZE, &aesKey)) < 0)
    {
        fprintf(stderr, "AES_set_encrypt_key: returned %d", res);
        return EXIT_FAILURE;
    }

    int i = 0;
    //for (int i = 0; i < 32; i += BLOCK_SIZE)
    {
        printf("ivec = ");

        for (int j = 0; j < BLOCK_SIZE; j++)
            printf("%.02hhx ", ivec[j]);

        putchar('\n');

        printf("input = ");

        for (int j = i; j < (i + BLOCK_SIZE); j++)
            printf("%.02hhx ", input[j]);

        putchar('\n');
        putchar('\n');
        putchar('\n');
        putchar('\n');

        AES_cbc_encrypt(input, outBuf, BLOCK_SIZE, &aesKey, ivec, AES_ENCRYPT);

        printf("outBuf = ");

        for (int j = 0; j < BLOCK_SIZE; j++)
            printf("%.02hhx ", outBuf[j]);

        putchar('\n');

        int res = write(outFD, outBuf, BLOCK_SIZE);

        if (res == -1)
        {
            perror("write");
            return EXIT_FAILURE;
        }
        else if (res < BLOCK_SIZE)
        {
            printf("Warning: unexpectedly wrote < %d bytes.\n", BLOCK_SIZE);
        }
    }

    if ((close(outFD)) == -1)
    {
        perror("close");
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

Build Command

gcc -O0 -ggdb test.c --std=c99 -lssl -lcrypto && ./a.out

Sample Output

ivec = 00 00 00 00 00 00 00 00 d0 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
input = 4d 69 64 6e 69 67 68 74 5f 4d 61 72 6c 69 6e 05 52 69 63 68 61 72 64 52 69 63 68 61 72 64 06 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 



outBuf = 81 ee 91 c0 9f f6 40 db 3c 6d 32 dd 5e 86 6f f8 4e 7b aa 15 38 36 b8 20 bc 04 bd 4f 6c 53 0e 02 72 c2 b7 e8 79 35 f2 b2 e1 c1 6e 1e 3b 1e 75 81 6a 56 43 d8 9d 9c 4c 1e 04 bd 99 29 3a 55 c9 a4 90 48 20 13 5e 51 4a 0c 4b 35 bc db da 54 f1 2b 66 f6 1b 1a 42 25 33 30 0e 35 87 9d 4b 1f d5 3a 5d 3a 8e 8c c8 48 c0 52 72 c0 4e b3 b8 f5 37 03 1c 87 15 61 3b 64 2b 06 5e 12 8f c7 b5 21 98 06

Why does the output buffer need to be one byte larger than the input buffer? Is that implementation-specific or just an AES thing? Also, point 4, why am I iterating over each byte? The for-loop uses increments of 16 to index `input` and passes it into the encrypt function. That's why output buffer is 16 bytes, just to test encrypting two lots of input in a row really, and why 16 bytes is hard coded. It's not meant to be a functional program, just a test. Thanks for testing for me. — Doddy, Jan 08 '16 at 21:59
**Correction**: The output is rounded up to the nearest multiple of the block size. My `+1` is incorrect, and unnecessary. I see what you mean now with the increment size of `16`. It seemed weird that you were effectively destroying the first set of encrypted data. Even if it's not a production program, you should get into the habit of consistent style guide and best-practise adherence. It'll make life easier. If this adequately answered your question, please consider marking this question as "accepted" via the checkmark. — Cloud, Jan 08 '16 at 22:09

Segfault in AES_cbc_encrypt

1 Answers1

Modified Code Listing

Build Command

Sample Output