31

I know how to download a file in this way: key.generate_url(3600)

But when I tried to upload: key.generate_url(3600, method='PUT'), the url didn't work. I was told: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I cannot find example code on the boto homepage for how to use the function generate_url(method='PUT'). Does anyone here know how to use it for uploading? How to set the params for the path of upload file?

djvg
  • 11,722
  • 5
  • 72
  • 103
michael.luk
  • 551
  • 2
  • 6
  • 11
  • Is that a new file? To create new file, you should use `POST`, not `PUT` – vartec Apr 06 '12 at 13:56
  • @vartec: What do you mean "a new file"? In my use case, sometimes I need to upload new key to a certain bucket, sometimes I need to overwrite the old key. So I think I need the code example for 'PUT' and 'POST'. – michael.luk Apr 06 '12 at 14:11

5 Answers5

50

I found some time to experiment with this and here's what I found.

>>> import boto
>>> c =boto.connect_s3()
>>> fp = open('myfiletoupload.txt')
>>> content_length = len(fp.read())
>>> c.generate_url(300, 'PUT', 'test-1332789015', 'foobar', headers={'Content-Length': str(content_length)}, force_http=True)
'http://test-1332789015.s3.amazonaws.com/foobar?Signature=oUARG45mR95utXsiQYRJNiCI4x4%3D&Expires=1333731456&AWSAccessKeyId=AKIAJOTCCJRP4C3NSMYA&Content-Length=16'

I was then able to use curl to PUT the file to that url like this:

$ curl --request PUT --upload-file myfiletoupload.txt "http://test-1332789015.s3.amazonaws.com/foobar?Signature=oUARG45mR95utXsiQYRJNiCI4x4%3D&Expires=1333731456&AWSAccessKeyId=AKIAJOTCCJRP4C3NSMYA&Content-Length=16"

This resulted in the file being uploaded to the bucket. So, it appears that it is possible. You might want to see if you can calculate the content-md5 value and include that in the headers but then you also have to figure out how to get curl to send that header, as well. Also, you should be able to make this work over HTTPS rather than HTTP but I haven't tried that.

garnaat
  • 44,310
  • 7
  • 123
  • 103
  • 2
    `headers` argument in this answer will not work now, it will cause 'signature mismatch` error when you upload. If you want to specify headers *in the response* when you GET the file later, you need to use `response_headers` and use the fields such as `response-content-type` etc. – MLister Jul 03 '15 at 19:22
  • Tested on Google Cloud Storage but failed. Would you mind taking a look at this question? Thanks. http://stackoverflow.com/questions/38938203/django-heroku-boto-direct-file-upload-to-google-cloud-storage – Randy Tang Aug 14 '16 at 07:38
37

Here's what it looks like in boto3(tested with version 1.2.3).

First, create a presigned url with s3.generate_presigned_url method:

>>> import boto3
>>> s3 = boto3.client('s3')
>>> s3.generate_presigned_url('put_object', Params={'Bucket':'YourBucket', 'Key':'YourKey'}, ExpiresIn=3600, HttpMethod='PUT')
u'https://s3-ap-northeast-1.amazonaws.com/YourBucket/YourKey?AWSAccessKeyId=AKIAXXXXXXXXXXXXXXXX&Expires=1451061671&Signature=%2FtyAyCd5vrp13p%2FqLdoPkox7yTM%3D'

PUT to S3 with a presigned URL

$ curl \
  --request PUT \
  --upload-file path/to/file \
  "https://s3-ap-northeast-1.amazonaws.com/YourBucket/YourKey?AWSAccessKeyId=AKIAXXXXXXXXXXXXXXXX&Expires=1451061671&Signature=%2FtyAyCd5vrp13p%2FqLdoPkox7yTM%3D"
quiver
  • 4,230
  • 1
  • 22
  • 20
  • how do you add your credentials i.e. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY? – CpILL Feb 11 '16 at 10:25
  • 1
    @CpILL You can pass credentials via environment variables. e.g., $ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-environment – quiver Feb 12 '16 at 13:57
  • 2
    Thanks for showing how to do in `boto3`. Currently migrating code from `boto` to `boto3` and wish their docs were clearer on all the changes. – maryokhin Aug 14 '16 at 01:08
  • I spent 21 hours on this problem until I finally saw this code. Thanks. The docs suck in a way that's unbelievable. Glad to find someone who's been through the same hell and went through the other side victorious. – JasonGenX Feb 08 '21 at 19:25
19

All the other answers assume the file will be uploaded with curl, which is really not convenient in most python scripts. In the following, a pre-signed url is generated with boto3 and the file is uploaded with the requests library:

session = boto3.Session(aws_access_key_id="XXX", aws_secret_access_key="XXX")
s3client = session.client('s3')
url = s3client.generate_presigned_url('put_object', Params={'Bucket': 'mybucket', 'Key': 'mykey'})

requests.put(url, data=open("/path/to/file").read())
Régis B.
  • 10,092
  • 6
  • 54
  • 90
  • Other solutions scattered across SO use 'rb' and post. they didn't work for me but this one did – Kermit Nov 11 '21 at 16:52
9

This is a follow up to garnaat's answer from Apr 6 '12.

I am generating a signed URL server side, where I have credentials, and I pass it to a client such that a client can directly upload content. I trust the client far enough to allow it to upload arbitrary sized files, but not enough to give it security tokens. I wanted to avoid having the client tell the server how large its content would be as part of the request. Hence my follow up answer.

I was able to get the signed url for the PUT method working without specifying content length in the headers or specifying force_http=True.

Using Boto 2.31.1: as in garnaat's answere:

>>> import boto
>>> c =boto.connect_s3()

then instead I used:

>>> temp_url = c.generate_url(seconds_available, 'PUT', bucket_name, s3_key)

this produced a url in the following form:

https://s3_location/bucket_name/s3_key?Signature=Ew407JMktSIcFln%2FZe00VroCmTU%3D&Expires=1405647669&AWSAccessKeyId=kM__pEQo2AEVd_Juz4Qq

I was then able to use curl to post a file:

>>> os.system('curl --request PUT --upload-file true_measure/test_files/test_file_w_content.txt "'+temp_url+'"')

I did have a very difficult time figuring this out because I usually use python requests to write tests and debug; however I get an authentication failure when I try to use it to put a file to one of these boto generated signed urls using requests. I haven't fully debugged this, but I suspect it is because requests is adding a few additional headers as compared to what curl produces.

I hope this follow up answer spares someone else the debugging pain I went through.

user3525596
  • 149
  • 2
  • 3
1

If you are using boto (not boto3), the only way I was able to get an upload to work was using generate_url_sigv4. Using vanilla generate_url caused the same error as reported in the original question. It is possible there is an AWS account setting I don't know about that controls which function works.

In a Python interpreter with boto 2.49.0 and requests 2.22.0:

import boto
import os
import requests
os.environ['S3_USE_SIGV4'] = 'True'
c = boto.connect_s3(host='s3.amazonaws.com')
url = c.generate_url_sigv4(3600, 'PUT', 'my-bucket-name', 'bucket-path/to/file.txt')
with open('file.txt') as f:
    resp = requests.put(url, data=f.read())

>>> resp
<Response [200]>

If you don't connect with a hostname, you will receive this error when generating the URL:

boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.

Related:
What does ''HmacAuthV1Handler' object has no attribute 'presign'' mean?

If you are using boto3, a presigned POST seems to be better-documented:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html#generating-a-presigned-url-to-upload-a-file

Kenny Trytek
  • 41
  • 3