Upload to s3 with curl using pre-signed URL (getting 403)

Question

I'm using curl to call into a Java ReST API to retrieve a URL. Java then generates a pre-signed URL for S3 upload using my S3 credentials, and returns that in the ReST reply. Curl takes the URL and uses that for upload to S3, but S3 returns 403 "The request signature we calculated does not match the signature you provided. Check your key and signing method."

Here is the code I'm using to generate the pre-signed URL:

public class S3Util {
    static final AmazonS3 s3 = new AmazonS3Client( new AWSCredentials() {
        @Override
        public String getAWSAccessKeyId() {
            return "XXXXXXX";
        }
        @Override
        public String getAWSSecretKey() {
            return "XXXXXXXXXXXXXX";
        }
    });
    static final String BUCKET = "XXXXXXXXXXXXXXXXXXXXXXXXXXX";

    static public URL getMediaChunkURL( MediaChunk mc, HttpMethod method ) {
        String key = ...
        //way in the future (for testing)...
        Date expiration = new Date( System.currentTimeMillis() + CalendarUtil.ONE_MINUTE_IN_MILLISECONDS*60*1000 );

        GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(BUCKET, key, method);
        req.setExpiration(expiration);
        req.addRequestParameter("Content-Type", "application/octet-stream");

        //this gets passed to the end user:
        return s3.generatePresignedUrl(req);
    }
}

and in curl, run from bash, I execute this:

echo Will try to upload chunk to ${location}
curl -i -X POST \
        -F 'Content-Type=application/octet-stream' \
        -F "file=@${fileName}" \
        ${location} || (echo upload chunk failed. ; exit 1 )

Among other things, I have tried PUT, and I have tried "Content-type" (lowercase T). I realize I'm missing something (or somethings) obvious, but after reading the appropriate docs, googling and looking at lots of similar questions I'm not sure what that is. I see lots of hints about required headers, but I thought the resigned URL was supposed to eliminate those needs. Maybe not?

TIA!

Update:

Just to be clear, I have tested downloads, and that works fine.

Java looks like:

GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(BUCKET, key, HttpMethod.GET);
req.setExpiration(expiration);

and curl is simply:

curl -i ${location}

Answer 1

I've been able to generate a pre-signed URL via C# and upload it thereafter via curl as expected. Given my tests I suspect you are indeed not using curl correctly - I've been able to upload a file like so:

curl -v --upload-file ${fileName} ${location}

The parameter -v dumps both request and response headers (as well as the SSL handshake) for debugging and illustration purposes:

> PUT [...] HTTP/1.1
> User-Agent: curl/7.21.0 [...]
> Host: [...]
> Accept: */*
> Content-Length: 12
> Expect: 100-continue

Please note, that --upload-file (or -T) facilitates PUTas expected, but adds more headers as appropriate, yielding a proper response in return:

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< x-amz-id-2: [...]
< x-amz-request-id:  [...]
< Date: Tue, 31 Jan 2012 18:34:56 GMT
< ETag: "253801c0d260f076b0d5db5b62c54824"
< Content-Length: 0
< Server: AmazonS3

Answer 2

when doing this with curl, you need to place the url in single quotes or else half of the query string gets chopped off (the part with the key/signature).

Answer 3

The way to generate the URL:

private static URL generateRUL(String objectKey, String ACCESS_KEY, String SECRET_KEY, String BUCKET_NAME) {
    AmazonS3 s3Client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY));
    URL url = null;

    try {
        GeneratePresignedUrlRequest request  = new GeneratePresignedUrlRequest(BUCKET_NAME, objectKey);
        request.setMethod(com.amazonaws.HttpMethod.PUT);
        request.setExpiration(new Date( System.currentTimeMillis() + (60 * 60 * 1000)));

        // Very important ! It won't work without adding this! 
        // And request.addRequestParameter("Content-Type", "application/octet-stream") won't work neither
        request.setContentType("application/octet-stream");

        url = s3Client.generatePresignedUrl(request ); 
    } catch (AmazonServiceException exception) { 
    } catch (AmazonClientException ace) { }

    return url;
}

The way to upload the file:

public int upload(byte[] fileBytes, URL url) {
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setDoOutput(true);
    connection.setRequestMethod("PUT");
    connection.setRequestProperty("Content-Type", "application/octet-stream"); // Very important ! It won't work without adding this!
    OutputStream output = connection.getOutputStream();

    InputStream input = new ByteArrayInputStream(fileBytes);
    byte[] buffer = new byte[4096];
    int length;
    while ((length = input.read(buffer)) > 0) {
        output.write(buffer, 0, length);
    }
    output.flush();

    return connection.getResponseCode();
}

Answer 4

Despite the fact that GeneratePresignedUrlRequest accepts an http method argument (and has a setMethod function), it appears to be unusable for anything but GET.

http://wiki.nercomp.org/wiki/images/0/05/AmazonWebServices.pdf states "The practice of signing a request and giving it to a third-party for execution is suitable only for simple object GET requests." Perhaps setting another method can be used for something, but apparently not this.

So, instead, I had to follow the instructions here:

http://aws.amazon.com/articles/1434?_encoding=UTF8&jiveRedirect=1

This is more complex, because the client is required to post a complete form, rather than just using a URL, and also means all that post info has to be communicated to the client separately, but it does seem to work.