Making conversion similar to BN_hex2bn + BN_bn2bin without openSSL in C

Question

I currently use openSSL to convert values from encrypted string to what I thought was a binary array. I then decrypt this "array" (pass to EVP_DecryptUpdate). I make the conversion like this:

    BIGNUM *bnEncr = BN_new();
    if (0 == BN_hex2bn(&bnEncr, encrypted)) { // from hex to big number
        printf("ERROR\n");
    }
    unsigned int numOfBytesEncr = BN_num_bytes(bnEncr);
    unsigned char encrBin[numOfBytesEncr];
    if (0 == BN_bn2bin(bnEncr, encrBin)) { // from big number to binary
        printf("ERROR\n");
    }

Then I pass encrBin to EVP_DecryptUpdate and decryption works.

I do this in many places in my code and now want to write my own C function of converting hex to binary array, which I can then pass to EVP_DecryptUpdate. I had a go at this and converted my encrypted hex string to an array of 0s and 1s, but turns out that EVP_DecryptUpdate won't work with that. From what I could find online, BN_bn2bin "creates a representation that is truly binary (i.e. a sequence of bits). More specifically, it creates a big-endian representation of the number." So this is not just an array of 0s and 1s, right?

Can someone explain how I can make the hex->(truly) binary conversion myself in C, so I would get the format that EVP_DecryptUpdate expects? Is this complicated?

You are basically asking for the answer to this SO question: [How to turn a hex string into an unsigned char array?](https://stackoverflow.com/q/3221170/1380680) (except for the spaces). — Reinier Torenbeek, Dec 13 '18 at 22:46
@ReinierTorenbeek I don't think so - I need to convert a hex string to a sequence of bits in big endian representation. Would that be the same as unsigned char array?... — Monday to Friday, Dec 14 '18 at 09:07

score 0 · Answer 1 · answered Dec 13 '18 at 22:38

It is unclear why you want this, and it's definitely not advisable to roll your own implementation of the conversion functions (they may stop working with any number of internal changes to OpenSSL) but if you're interested in what it looks like:

static int bn2binpad(const BIGNUM *a, unsigned char *to, int tolen)
{
    int n;
    size_t i, lasti, j, atop, mask;
    BN_ULONG l;

    /*
     * In case |a| is fixed-top, BN_num_bytes can return bogus length,
     * but it's assumed that fixed-top inputs ought to be "nominated"
     * even for padded output, so it works out...
     */
    n = BN_num_bytes(a);
    if (tolen == -1) {
        tolen = n;
    } else if (tolen < n) {     /* uncommon/unlike case */
        BIGNUM temp = *a;

        bn_correct_top(&temp);
        n = BN_num_bytes(&temp);
        if (tolen < n)
            return -1;
    }

    /* Swipe through whole available data and don't give away padded zero. */
    atop = a->dmax * BN_BYTES;
    if (atop == 0) {
        OPENSSL_cleanse(to, tolen);
        return tolen;
    }

    lasti = atop - 1;
    atop = a->top * BN_BYTES;
    for (i = 0, j = 0, to += tolen; j < (size_t)tolen; j++) {
        l = a->d[i / BN_BYTES];
        mask = 0 - ((j - atop) >> (8 * sizeof(i) - 1));
        *--to = (unsigned char)(l >> (8 * (i % BN_BYTES)) & mask);
        i += (i - lasti) >> (8 * sizeof(i) - 1); /* stay on last limb */
    }

    return tolen;
}

Right... This looks complicated... about internal changes to OpenSSL - you mean changes to the decryption functions? — Monday to Friday, Dec 14 '18 at 09:08

Reinier Torenbeek · Accepted Answer · 2018-12-14T15:30:32.943

BN_bn2bin "creates a representation that is truly binary (i.e. a sequence of bits). More specifically, it creates a big-endian representation of the number." So this is not just an array of 0s and 1s, right?

The sequence of bits mentioned here is represented as an array of bytes. With each of those bytes containing 8 bits, this can be interpreted as an "array of 0s and 1s". It is not an "array of integers that have the value 0 or 1", if that is what you are asking.

Since you are unclear of the workings of BN_bn2bin(), it helps to just analyze the end result of your code snippet. You could do that like this (omitting any error checking):

#include <stdio.h>
#include <openssl/bn.h>

int main(
    int argc,
    char **argv)
{
    const char *hexString = argv[1];

    BIGNUM *bnEncr = BN_new();
    BN_hex2bn(&bnEncr, hexString);
    unsigned int numOfBytesEncr = BN_num_bytes(bnEncr);
    unsigned char encrBin[numOfBytesEncr];
    BN_bn2bin(bnEncr, encrBin);
    fwrite(encrBin, 1, numOfBytesEncr, stdout);
}

This outputs the contents of encrBin to the standard output, which is never a nice thing to happen, but you can then pipe it through a tool like hexdump, or redirect it to a file for analyzing with a hex editor. It looks like this:

$ ./bntest 74162ac74759e85654e0e7762c2cdd26 | hexdump -C
00000000  74 16 2a c7 47 59 e8 56  54 e0 e7 76 2c 2c dd 26 |t.*.GY.VT..v,,.&|
00000010

Or, if you do want to see those 0s and 1s:

$ ./bntest  74162ac74759e85654e0e7762c2cdd26 | xxd -b -c 4
00000000: 01110100 00010110 00101010 11000111  t.*.
00000004: 01000111 01011001 11101000 01010110  GY.V
00000008: 01010100 11100000 11100111 01110110  T..v
0000000c: 00101100 00101100 11011101 00100110  ,,.&

This shows that your question

Can someone explain how I can make the hex->(truly) binary conversion myself in C, so I would get the format that EVP_DecryptUpdate expects? Is this complicated?

is essentially the same as the SO question How to turn a hex string into an unsigned char array?, like I commented.

Making conversion similar to BN_hex2bn + BN_bn2bin without openSSL in C

2 Answers2

Linked