Forward Slash in Base 64 Alphabet Causing Issues with Generating Hash

Question

I am building a library to communicate with the back end that drives our iPad client.

The basic Structure for all requests is:

{
    "metaData":{ JSON Object that is not important for this question },
    "requestData":
        {
            "nonce":"random string",
            "params":"JSON string containing request data"
        }
     "checksum":"hash of JSON string representing value of requestData",
     "connectionString":"someIdentifier"
}

For the specific case where I am having trouble is when requestData has the following structure:

{
    "requestData":
        {
            "nonce":"random string",
            "params":
                "{\"userId\":1,\"formData\":\"encrypted string, then Base 64 encoded on iPad\"}"
        }
    "checksum":"hashed value of string representation of requestData"
}

An example of the string (as printed by NSLog) used to generate the checksum is:

{"params":"{\"groupId\":3,\"formData\":\"SExvR0J1ZkJSQkhObU5xZkiEXdBede2moVN3LtMDZlxcXYVj7Uz!BFdiQC9SwxIhrrcGv2GtWJzjqMhHzdFDZW568tbnLTKQ9931efrpjtvqlK9mudInXj0FQdBLY0M6f9zBlLu6TcQ7sA6AD15DF0HyUPIi4fnc90ZV7omGqRpyI412aGSpDPJEbCUBSY5WMUFJqRstyK1+Qo0vmN8uMprztDIyEFufP24DHHtYZHVAic8Sg8CxbsUTTYDgDc!0ASQwahEgy1sWkMP!BVpK8VU7quXDdIJrxbSNL7OO4tsJrHIXyhhK7ZUNKMaZX+fBSdw6DbNtTM86K0X4NSRXPVLE0EAklAJ2OpMDBsoz9k!jhCba5gRXY7r48USpsMyyj1v8SsDKn58FsvDxsdCrPY77KmIX3Icy!n3iA!lBfc3ol6c90wkwPSqNvnO7uRDYrfbP1c0zRYSXbLTQvHLLdfAWKariCKtNg6YAXNfgQ6lWFRXce8flHgUz6E7rkt9tjc9i4K+EjcL10H+E3AGkidYPGtQOm1vey!M8oineM!Cgg3VcvNCv!yN90iq3T+tqI0ivvBnh+1aCw2H90tnNm8Gi+XCrIdhORN3QjSkkNbpfoSCLoIkuBmXlNuTskaJ4nnV3kHrmU!4hYMeZIIZ8OnZWPpU47xJi!kh3MDdI2c+WorT+y+M5XwcQO6jGv3tXyRVBy!ne+sSnU!InISm7x1VQjJLmjULMnqxRDoZatBsofxICJysEUaDJvgwZasMJpQk1zyrPraBWBJ0lVVaWhH5OTi6U0!hHNVs5Xf+H23JxmPpUNWqNvsAGfnTfY!kSoiLoSxEocICK8zsJFMc69101DNAanayf!MjFFDeFRlzpKhcRON7cxDfvBdSoc9hL1lcMzFbLemrL1w8jNNMfKlY7QDZ5ebOERJMjY0!o8znlxOa0ViuJ++O7+QrT!mGdSQYGh3NJ3MK1IdJkXuFpY!guyXOgohTsqcD0DZSk84OsI76L18snFvs4qMHw9SUf3l0jWPxbTYimmlM3DVUR7Sn7xOsGmQGcwpGK1tinlIDA+w8Ci+CLWESsjZ5QDQCr\",\"internalFormId\":\"MTN13511759141\",\"userId\":1,\"code\":\"\",\"_queuedSubmission\":false,\"groupName\":\"LDMAdmin\",\"saleType\":\"1\",\"formTableName\":\"myTableName\",\"emailName\":\"default\"}","nonce":"XqfK9Nxwuggw4m"}

When I examine the server logs, I find the checksum generated on the device does not equal the checksum generated on the server (in PHP, case sensitivity is not the issue), thus authentication fails.

An example of the structure that fails is:

PageManager.m - 905 -> Final Request:
{
    "checksum":"9D51170D1510C4081936870D11E96C869DB26B895393B9C14B2A6BC3C1F10F23",
    "connectionString":"testBed",
    "metaData":
        {
            "accessToken":"myAccessToken",
            "appId":"myAppID",
            "deviceId":"1X:1X:1X:1X:1X:1X",
            "groupId":3,
            "groupName":"LDMAdmin",
            "timestamp":1351018002.780379,
            "userId":1,
            "useragent":"iPhone OS",
            "username":"admin"
        },
    "requestData":
        {
            "nonce":"1iezcBdjbE",
            "params":"{\"groupId\":3,\"formData\":\"Tnh3dWdndzRtRmdxSFNmN/NXCIQSukpx3+mhmbNQh0PTGbLlEFoDinyrq3wRJGZ+8sQ/+xcjS4cU7evluipxqQDZIOvp4ZcoDnxTPeqBZJrG/bq5FHR6PVCYK2DaLHfj025z/H3RM8dUEoWcrTLqSUcW+E7Mfl8ZCApqJMxSa8eYYqLT7tm7r1SC+bjXNOQZLTC2laFhihQ5hLKqFFnO/z3AlUYAAUhKKD1lWIipnJUUNoyHdWuuOobMSS1ZZP5f5f+RTFsmGZUDe6qX6h2cjIQ2+VGPIsP//gqwO4iDx/FdHD+xrjCyEgL2Va/m/Z+ANxCr3DN2o2Jnwg8B8QycFN2tGrgusseAgoa9Ng9LRgooZW+KuECWDhorHzvuv2rOlhOskymj4XTu8890ZMJbcr1Ic6zwztm82R1qKaoy1o6gIbUNtVZFSqUlP8TO7mWHKr3Y8Awn7ih9HzSOg1486EDL4OjfOR9J2pw1jbK7ZJb7LxzrWFgoyrwDBAw3q7PrV4Ml9ngI6oXOh3veAq/wulyBOdF46n7evqIkAKg4FYdvzmFKd2bgOpxwBlAI7vL2IiC4v8GXI5977SkPPEKUZHXWmfrgr/VzF79gIxJDqV9N0ceAcgY8bWbBXf7DLd9H82obFa60yZBo5/MBjq9SNuD08vJEEauVGs4wfDr9+xzsr3z+plqxAejODdxKfF48Ra21L8Xozozv5papTP9cpGVU11mCWj+no5gtM0VQKRB7IQcpDWjQgQyThN2aoE06ecA2gY5SSXN0XHVRw5OKM0/rlNIuMiqow5wqHLl41IzDSF2HuJKj06Lv8t5CLLOd9rkOjYw6w8SrbsZeG5jwagJkyQ0UuKu+PIoIc2DJnUWDC5iqlb0TO9nPDNFKad+MYlfgDR0CxR+3ddkqWNBNSW5rsh5QZDlJHDjhQFLkuqiiRAnMvKOcbqAnXIZ9EuAo/DkcmtGPHkEyEaA2cb3mXysBP49jhY0m/qinloza+j3d7Kb/Fu35U929fOxH6+W+5oZv/r+a9KvkDhPoRwiFouVwTtTOwbjVDT+NEg2OUfDaEYbQ/RbM7i6X+XjSkZMLYsRs1Q9CwdBabY860uBNFQ==\",\"internalFormId\":\"MTN13510178111\",\"userId\":1,\"code\":\"\",\"_queuedSubmission\":false,\"groupName\":\"LDMAdmin\",\"saleType\":\"1\",\"formTableName\":\"myTableName\",\"emailName\":\"default\"}"
        },
    "url":"http://myurl.com/myAction"
}

The formData is generated by creating a JSON string of the desired data, encrypting this string, and encoding the resulting NSData object into a base 64 string. The checksum is generated by taking the hash of the JSON string representing the requestData value (with SHA256).

requestData is converted to a JSON string, and hashed with:

+(NSString *)hmacSHA256:(NSString *)string withKey:(NSString *)key {
    NSString *hash = 0;
    NSData   *hmac = 0;

    NSMutableString *temp = [[NSMutableString alloc] initWithString:@""];

#ifdef DEBUG
    NSLog( @"%s - %d -> Values:\nString:  %@\nKey:  %@", __FILE__, __LINE__, string, key );
#endif


    const char *cKey  = [key cStringUsingEncoding:NSASCIIStringEncoding];
    const char *cData = [string cStringUsingEncoding:NSASCIIStringEncoding];

    unsigned char cHMAC[CC_SHA256_DIGEST_LENGTH];
    unsigned char *digest;
    unsigned int  dLength;

    CCHmac( kCCHmacAlgSHA256, cKey, strlen(cKey), cData, strlen(cData), cHMAC );

    hmac = [[NSData alloc] initWithBytes:cHMAC length:sizeof(cHMAC)];

    digest  = (unsigned char *)[hmac bytes];
    dLength = hmac.length;

    for ( int i = 0; i < dLength; ++i )
        [temp appendFormat:@"%02X", digest[i]];

    hash = [[NSString alloc] initWithString:temp];

    return hash;
}

The PHP script uses a stock function to generate the hash:

protected static function _generateChecksum($data, $key, $output)
    {
        $jsonData = json_encode($data);
        log_message('error', 'encoded string is: '.$jsonData);
        $checksum = hash_hmac('sha256', $jsonData, $key);
        //log_message('error', 'Checksum pre encode: '.$checksum);
        // if($output == true)
        // {
        //     $checksum = base64_encode($checksum);
        // }
        return $checksum;
    }

The hash is compared to the hash sent in the request.

An example of the server log:

ERROR - 2012-10-23 21:31:16 --> Encoded Checksum recieved is: 14F03A1DCAEA9DBBC7EC8CA1D666D89C391760AA246C91B52164D526AC83E5E5
DEBUG - 2012-10-23 21:31:16 --> Model Class Initialized
DEBUG - 2012-10-23 21:31:16 --> Model Class Initialized
DEBUG - 2012-10-23 21:31:16 --> Database Driver Class Initialized
ERROR - 2012-10-23 21:31:16 --> encoded string is: {"params":"{\"groupId\":3,\"formData\":\"SE5tTnFmWHFmSzlOeHd1ZyYzHN9y\/aJSbsqEn6X1TypGwWcXtXGpW4ODrCDVwZyIjhq3oOeZ4C6hGCDFGqHsa5hhNXxeerWLG5SyvfksCTG1+GCvWFwMx0CzZwOAfJRwSoCBCaeZ\/pivs3dHQS22SEbWn6+2e2vayeap7mxvZZw9Jrl\/c4dGFAiNqQB5pQbNO661AqbJWDAHCS8EWBhXXsd0SbTHlZAip4H0MdlF2rnElCVfHlc01RcuJNXLF3NJvfjY9m4sXmI3BAED0c0C\/i0Uw2M6pe4iDJv\/OvOI0NVS8RKbRbjhTo3oktAmNttfKTG6xp0wMhbANppuoo4QY3XwQ5BKUjqhmr5kx8j0RTmebcTCmxsC9h1dqjHYnf1JnZDFATkVsKnn\/Ela1wSjhGL7uP6jl3r4xDGKGPWDj0E3iAPNN56pmJxzyQrHOOqUzGbmvU3qj7Ul039IGYZTzn74VUkWi3JsxJH+kU9iWSvuC+YoOHcf\/0OFn1PqBoDHjDTbN+3HV8wwSqrVFJ6z9RX4MwRfffVgKl8xL2hHqBnegjvyd65KbZbSd\/3OrEBeL0dAjuARiPioNTpjzwga4chFRA471gweLT+cKweZBXZMYll36sNqulIBzbCmqbndDk63Id9iSrs9\/fQVWUA7RJDudnAxvQPs8gTznp9Dz1SomyY4ONYrJ9EticAEnUEjF2sCdejYlgu61a3Zss19m+MzgEhxkwmRwttsRbFfNK44wP\/wB2FgdfjsY94nHpJ+6lPEZtRmWpYtNVxQMVC6mMde6CbSEem71byIiN424baPImtNIfF+bKl6BKxyEl7BhI3z25NXaKyfaflzxGY8Yvdg0f73SfT3omPP3KxdudFgJrQ6eiO2AXt5L6lPjezjRr17R6hTUNmYwvZ3C5S0zoY7ynmCHebeiNlavVepUBkn0Iu6w\/qKDJ5wr80n7XX3EXuo1ODCC5aCjOSr+gSS9eVm0\/IBQNF\/ec4kjI29LRyrOFOS\/2poHY9XzyagVURiwi101a0yPETRRsC8n4B\/XFmOFQ0VcCQNgTuXute52fsccxB3DkV7ixBbQ8mt6o2XDWGk2HnrDwmRuNX87rBHow==\",\"internalFormId\":\"MTNC13510277491\",\"userId\":1,\"code\":\"\",\"_queuedSubmission\":false,\"groupName\":\"LDMAdmin\",\"saleType\":\"1\",\"formTableName\":\"myTableName\",\"emailName\":\"default\"}","nonce":"gw4m"}
ERROR - 2012-10-23 21:31:16 --> Encoded Checksum expected is: 8d999d76e48907905e701da3ccdbccb4061d05ed5a7c18b58507b6e6352fb1f5

However, other components of the application use the same structure (minus the base 64 encoding) behave as designed and expected. The checksum is generated using the same method and key as the example above. Example:

{
    "requestData":
        {
            "nonce":"random string",
            "params":"{\"latitude\":37.7,\"longitude\":-122.4}"
        }
    "checksum":"hashed value of the string representation of requestData"
}

I have no idea why the PHP script on the server would be generating a different hash in the failed case mentioned above. Has anyone encountered this type of problem before?

UPDATE:

I am using SBJson for encoding and decoding my data structures into JSON strings (link).

UPDATE 2:

Base on the discussion that has occurred so far, there is no guarantee the order in which a JSON object (or in my case an NSDictionary) is serialized to a JSON string. But if that were the case, I do not understand why cases where data is sent in clear text, the same hash is generated on the server, and in the case where some data is in the form of base 64 string, a different hashes are generated between the server and the client.

The only solution, which is probably not right solution, is to remove / from the base 64 alphabet (I replaced it with !). I am pretty sure this will cause problems when trying to decode the string on the server side.

At this point, I would like to understand how a forward slash could be causing this problem.

I believe nobody else will get an idea unless you show us the code that calculates the hashes. Obviously something goes wrong there. — Sven, Oct 23 '12 at 21:41
Well, the first error might be that you actually decode the string somewhere, then encode it again to JSON before checking the hash. This might ultimately change the string, destroying the hash. Second thing is: I cannot see if you accidentially pass the checksum value into your PHP hash. It should be removed of course - but on the string level. — Sven, Oct 23 '12 at 22:38
The first step in debugging would be to check if both implementations of hashing deliver the same result when used on a constant string (i.e. the same bytes!). — Sven, Oct 23 '12 at 22:40
@Sven I have checked the server code, it is decoding elements in proper order. Also, if I send the data as clear text, authentication passes, so I am thinking the error is in the encryption. — Mike D, Oct 24 '12 at 15:20
@Sven Using a forward slash in the base 64 encoded string caused unexpected behaviour. But this will probably be cause decoding issues. — Mike D, Oct 24 '12 at 16:33

score 0 · Answer 1 · answered Oct 23 '12 at 21:39

0

There's no such thing as "the JSON string representing such-and-such". If you have a data structure in memory and use your platform's default JSON serializer on it, you have no guarantee that it will produce the same string as another platform's default JSON serializer -- or even that it will produce the same string as the JSON serializer you use will produce tomorrow, or on Fridays in leap years.

JSON gives serializers considerable latitude in where to place whitespace, how to format numbers, which order to emit object fields, and so forth. It cannot be used as a basis for hashing the underlying abstract dataset.

answered Oct 23 '12 at 21:39

hmakholm left over Monica

23,074
3
51
73

You can always hash a string, but adding the hash value to this very string afterwards will change it. If you do not reverse this process properly, you will not get back the original string that was used for hashing. – Sven Oct 23 '12 at 21:53
@Sven: The problem is that you may get _different_ strings as JSON representations of the _same_ JSON object. If you hash different strings, you will generally get different hashes. – hmakholm left over Monica Oct 24 '12 at 19:14

score 0 · Answer 2 · edited May 23 '17 at 11:55

0

It was the forward slash that was in the encoding table, and somehow, this was causing a conflict when generating the hash in the php code.

The original table in my Objective C code:

//64 digit code
static char Encode[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";

Which I had to replace with:

//64 digit code
static char Encode[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+!";

This may be only a temporary solution, since this SO answer states that forward slashes are acceptable in a base 64 encoded string. Simply replacing the forward slash solved the issues for now.

edited May 23 '17 at 11:55

Community

1
1

answered Oct 24 '12 at 16:31

Mike D

4,938
6
43
99

That doesn't address the real problem, which is that _you have no right to expect that two different JSON serializers produce identical string representations in the first place_. You have no right to expect that _one_ JSON serializer produces the same string representation of your data on Tuesday as it did Monday. That's not merely theoretical; it is quite plausible for a serializer to select the representation order of fields in an object according to the unpredictable sorting behavior of the implementation's internal hash table representation of objects. – hmakholm left over Monica Oct 24 '12 at 19:17
In your example, the serializer is clearly not emitting the fields in sorted order -- it has "groupId" coming before "formData" and "params" before "nonce". – hmakholm left over Monica Oct 24 '12 at 19:19
@HenningMakholm If that were the case, then why would unencrypted strings pass authentication? – Mike D Oct 24 '12 at 19:45
x @Mike: Not sure what you mean by "encryption" in this context. – hmakholm left over Monica Oct 24 '12 at 20:21
@HenningMakholm Sorry, why would plain text pass authentication vs the exact same data sent as base 64 encoded string? – Mike D Oct 24 '12 at 20:24
x @Mike: The data you base64 encode is the bytes that come out of the hash function, right? How do you transmit them in JSON (and figure out that they "pass authentication") if _not_ as something like base64 encoding? – hmakholm left over Monica Oct 24 '12 at 20:26
@HenningMakholm If you look at my example that fails, the value for "formData" is encrypted data encoded as a base 64 string. I know this encryption and encoding works, it is used in our current service. The new part is generating a hash of the "requestData." If I skip the encryption and encoding, the expected hash is generated on both the client and the server. – Mike D Oct 24 '12 at 20:32
x @Mike: Still not sure what you mean by "encryption". But feel free to keep believing that JSON serializers will produce deterministic results on a string level; you're the one who will get to deal with the stream of bugs that will result. – hmakholm left over Monica Oct 25 '12 at 10:53

Forward Slash in Base 64 Alphabet Causing Issues with Generating Hash

2 Answers2