AWS SDK iOS 2.0 S3 upload: add correct md5

Question

I am trying to upload a file to S3 using the new AWS SDK for iOS 2.0. The upload works fine as long as I don't set a contentMD5 in the request.

First, I create a filepath and a URL:

NSString *tempFilePath = [NSTemporaryDirectory() stringByAppendingPathComponent:@"s3tmp"];
NSURL *tempFileURL = [NSURL fileURLWithPath:tempFilePath];

Next, I create the request:

AWSS3TransferManagerUploadRequest *uploadRequest = [AWSS3TransferManagerUploadRequest new];
uploadRequest.bucket = S3_BUCKETNAME;
uploadRequest.key = s3Key;
uploadRequest.body = tempFileURL;

Then I create the md5. Conveniently, there is a github project here: https://github.com/JoeKun/FileMD5Hash which is creating the md5 from a file. However, I cross checked with bashs md5 and it's returning the same md5 string. Additionally, if I don't set contentMD5 in the request, the upload succeeds and in the console the same md5 string is shown as eTag. So I think the md5 calculated in the next line is correct.

NSString *md5 = [FileHash md5HashOfFileAtPath:tempFilePath];

Finally, I add the md5 to the uploadRequest:

uploadRequest.contentMD5 = md5;

and start the upload:

[[transferManager upload:uploadRequest] continueWithBlock:^id(BFTask *task) {
NSError *error = task.error;
if (error) {
NSDictionary *errorUserInfo = error.userInfo;
NSLog(@"Error %@: %@",[errorUserInfo objectForKey:@"Code"],[errorUserInfo objectForKey:@"Message"]);
dispatch_sync(dispatch_get_main_queue(), ^{
[weakSelf uploadFinishedUnsuccessful];
});
}
else {
NSLog(@"Upload success for file \n%@ to \n%@/%@",[tempFileURL absoluteString],S3_BUCKETNAME,s3Key);
dispatch_sync(dispatch_get_main_queue(), ^{
[weakSelf uploadFinishedSuccessful];
});
}
return nil;
}];

This always returns an error: Error InvalidDigest: The Content-MD5 you specified was invalid.

So I tried wrapping the md5 into base64, using the builtin method of iOS:

NSString *base64EncodedString = [[md5 dataUsingEncoding:NSUTF8StringEncoding] base64EncodedStringWithOptions:0];

I cross checked this with another base64 library. It returns the same base64 string, so I think the base64 string is correct. I tried setting this as contentMD5:

uploadRequest.contentMD5 = base64EncodedString;

I get the same error: Error InvalidDigest: The Content-MD5 you specified was invalid.

Any idea what I am doing wrong?

Thanks for any reply!

score 4 · Accepted Answer · answered Oct 16 '14 at 04:53

4

You need to base64-encode the binary representation of the MD5 hash... not the hex representation, which is what it sounds like you may be doing.

The resulting value will be somewhere the neighborhood of 24 characters long, if encoded correctly... and twice that long if done incorrectly.

answered Oct 16 '14 at 04:53

Michael - sqlbot

169,571
25
353
427

Thank you very much for your help, Michael! I create an NSData object like this: [md5 dataUsingEncoding:NSUTF8StringEncoding] --isn't that the binary representation of the md5? Then I encode that NSData object into base64. Is that wrong? – marimba Oct 16 '14 at 08:13
Objective C is not an area of expertise for me. S3 is, and I've implemented code before that correctly calculates the Content-MD5 that S3 expects. Your description of your md5 being the same as the etag led me to suspect that you're starting out with the right value, so the most likely issue is that you're encoding to base64 from the wrong starting point. What's an example of one md5 sum and the corresponding b64-equivalent as calculated by your code? I'll see what value I come up with. – Michael - sqlbot Oct 16 '14 at 10:23
The md5 is: "28e01cf6608332ae51d63af3364d77f2" (local and as eTag on S3). The resulting base64 is "MjhlMDFjZjY2MDgzMzJhZTUxZDYzYWYzMzY0ZDc3ZjI=" - 48 chars, you're right!! So what is the difference between "correctly" and "incorrectly"? – marimba Oct 16 '14 at 11:17

score 4 · Answer 2 · edited May 23 '17 at 12:22

Ok, Michaels comments got me in the right direction. After a lot of reading back and forth I finally understood what he tried to tell me. For anyone else struggling to grasp the concept, I try to explain: An md5 string like "28e01cf6608332ae51d63af3364d77f2" is the hexadecimal representation of a 16 byte digest. That means, each 2 characters of those 32 characters represent 1 byte.

28 = First byte, e0 = Second byte, 1c = third byte, and so on, until you have 16 bytes.

The content-md5 header expects the base64-encoded 16 bytes, not the hexadecimal representation. The otherwise very convenient method

[FileHash md5HashOfFileAtPath:tempFilePath]

returns only this hexadecimal representation as NSString. So I could either dig into extracting the bytes before the string conversion is done, or re-convert the string into NSData. I chose the latter with a code snippet I found here:

//convert the md5 hexadecimal string representation BACK to the NSData byte representation
    NSMutableData *md5Data= [[NSMutableData alloc] init];
    unsigned char whole_byte;
    char byte_chars[3] = {'\0','\0','\0'};
    int i;
    for (i=0; i < [md5 length]/2; i++) {
        byte_chars[0] = [md5 characterAtIndex:i*2];
        byte_chars[1] = [md5 characterAtIndex:i*2+1];
        whole_byte = strtol(byte_chars, NULL, 16);
        [md5Data appendBytes:&whole_byte length:1];
    }

    //base64-encode the NSData byte representation
    NSString *base64EncodedString = [md5Data base64EncodedStringWithOptions:0];

    uploadRequest.contentMD5 = base64EncodedString;

This base64encodedString returns a success response, finally! Thanks Michael!

I have added md5 string to S3PutObjectRequest as this s3PutObjectRequest.contentMD5 = base64EncodedString; s3PutObjectRequest.httpMethod = @"PUT"; But when i try retrieve eTag from S3PutObjectResponse i get ErrorCode:BadDigest, Message:The Content-MD5 you specified did not match what we received. any reason ?? i have put my question here please add a comment i you have any idea about this reason http://stackoverflow.com/questions/29847564/request-checksum-from-amazon-s3-in-ios — Mr.G, Apr 27 '15 at 09:48

AWS SDK iOS 2.0 S3 upload: add correct md5

2 Answers2