Wrong SHA1 from PHP & CF according to Amazon S3

Question

Amazon consistently generates a different hash than PHP or CF, which causes a persistent "SignatureDoesNotMatch" error.

According to the docs, GET requests [without REST headers] are signed as follows:

Signature = URL-Encode( Base64( HMAC-SHA1( SecretAccessKey, UTF-8-Encoding-Of( StringToSign ) ) ) );

StringToSign = HTTP-VERB + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Expires + "\n" +
    CanonicalizedAmzHeaders +
    CanonicalizedResource; 

The example data:

• SecretAccessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
• Content-MD5 and Content-Type: (optional - skipped)
• CanonicalizedAmzHeaders: (no headers - skipped)
• Resource: johnsmith.s3.amazonaws.com/photos/puppy.jpg
• CanonicalizedResource: /johnsmith/photos/puppy.jpg

Two examples are provided:

1. Expires 1175139620; Signature: rucSbH0yNEcP9oM2XNlouVI3BH4%3D
2. Expires 1141889120; Signature: vjbyPxybdZaNmGa%2ByT272YEAiv4%3D

To recreate this (CFHMAC from here):

// PHP
$expires = 1175139620;
$SecretAccessKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
$StringToSign = "GET\n\n\n$expires\n/johnsmith/photos/puppy.jpg";
$signature = urlencode( base64_encode( hash_hmac('sha1',  utf8_encode($StringToSign), $SecretAccessKey, true)));

// ColdFusion
<cfset LF = chr(10)>
<cfset expires = 1141889120>
<cfset SecretAccessKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY">
<cfset StringToSign = "GET#LF##LF##LF##expires##LF#/johnsmith/photos/puppy.jpg">
<cfset signature = URLEncodedFormat( CFHMAC(StringToSign, SecretAccessKey))>

EXCEPT that $signature as returned by both languages is:

1. Expires 1175139620; Signature: NpgCjnDzrM%2BWFzoENXmpNDUsSn8%3D
2. Expires 1141889120; Signature: fScKGHCDI0NY5E7CYp9Vc8VKMbY%3D

We have been careful of these gotchas that others have mentioned:

1. hash_mac has a third argument, raw, which must be set to true.
2. The order of the stringToSign and key in the S3 psuedocode should be reversed.
3. The entire stringToSign must be on one line (so as not to create extra newline characters).

EDIT: Updated the newlines in the CF code based on Leigh's answer; now the CF matches the PHP.

I am obviously doing something wrong, but can't figure out what.
[I have heard it quipped that Amazon S3 would have been called CSS - "complicated storage service", but the name was already taken!]

Help, please!

Answer 1

Would this help?

<cffunction name="getRequestSignature" access="private" output="false" returntype="string">
    <cfargument name="verb" type="string" required="true" />
    <cfargument name="bucket" type="string" required="true" />
    <cfargument name="objectKey" type="string" required="true" />
    <cfargument name="dateOrExpiration" type="string" required="true" />
    <cfargument name="contentType" type="string" default="" />
    <cfargument name="contentMd5" type="string" default="" />
    <cfargument name="canonicalizedAmzHeaders" type="string" default=""
        hint="A newline-delimited list of headers, in lexographical order, duplicates collapsed, and no extraneous whitespace.  See Amazon's description of 'CanonicalizedAmzHeaders' for specifics." />
    <cfscript>
        var stringToSign = "";
        var algo = "HmacSHA1";
        var signingKey = "";
        var mac = "";
        var signature = "";

        stringToSign = uCase(verb) & chr(10)
            & contentMd5 & chr(10)
            & contentType & chr(10)
            & dateOrExpiration & chr(10)
            & iif(len(canonicalizedAmzHeaders) GT 0, de(canonicalizedAmzHeaders & chr(10)), de(''))
            & "/" & listAppend(bucket, objectKey, "/");
        signingKey = createObject("java", "javax.crypto.spec.SecretKeySpec").init(variables.awsSecret.getBytes(), algo);
        mac = createObject("java", "javax.crypto.Mac").getInstance(algo);
        mac.init(signingKey);
        signature = toBase64(mac.doFinal(stringToSign.getBytes()));

        return signature;
    </cfscript>
</cffunction>

Completely stole it from here: http://www.barneyb.com/barneyblog/projects/amazon-s3-cfc/

:)

Answer 2

(May as well post this since I had already written it up .. :)

Two problems I can see

1. The date needs to be formatted a specific way
2. You need to use a LF rather than a literal "\n"

The result below matches that in Authentication Examples ie bWq2s1WEIj+Ydj0vQ697zp+IXMU=. Note: I used the hmacSHA1 function from here, but changed it use getBytes("UTF-8)

Code:

    <cfset newLine = chr(10)>
    <cfset secretAccessKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY">
    <cfset stringToSign = "GET#newLine##newLine##newLine#Tue, 27 Mar 2007 19:36:42 +0000#newLine#/johnsmith/photos/puppy.jpg">
    <cfset signature = hmacSHA1(secretAccessKey, stringToSign)>
    <cfset finalSignature = URLEncodedFormat(binaryEncode(signature, "base64"))>
    <cfoutput>finalSignature = #finalSignature#</cfoutput>

****EDIT 1:**

Something is fishy. Most all of the examples on that page match up. But REST Authentication Example 3: Query String Authentication Example here shows a different key and string that produce the signature vjbyPxybdZaNmGa%2ByT272YEAiv4%3D. If you use those values in CF you do get the same signature. So I am wondering if it might just be a documentation error?

     <cfset secretAccessKey = "OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV">
     <cfset stringToSign = "GET#newLine##newLine##newLine#1141889120#newLine#/quotes/nelson">

** EDIT 2:

I am pretty sure the REST examples are wrong. A search turned up this link containing yet another sample key. If you substitute that in the CF code, the signature is what you expected: rucSbH0yNEcP9oM2XNlouVI3BH4%3D.

    <cfset secretAccessKey = "uV3F3YluFJax1cknvbcGwgjvx4QpvB+leU8dUj2o">
    <cfset stringToSign = "GET#newLine##newLine##newLine#1175139620#newLine#/johnsmith/photos/puppy.jpg">