0

Given a url of a downloadable file, such as:

http://zannee-docs.s3.amazonaws.com/rockefeller-summary.pdf

Is there a way to POST / PUT this directly to S3? Or is it necessary that I do:

# download the file
requests.get(url, stream=True)

# save the file locally

# upload the local file to S3
s3.upload_file()

If there is a way to upload it directly to S3 (without having to download it and re-upload it to S3), how would I do this?

David542
  • 104,438
  • 178
  • 489
  • 842

2 Answers2

3

Yes, but only if the source URL is also hosted in S3, and your S3 key can access it. (The documentation doesn't say this explicitly, but I think the source object needs to be stored in the same region as well.)

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html

Since you're using boto, the specific API you'll want is boto.s3.key.Key.copy().

  • Thanks, so if the object is owned by someone else, it wouldn't be possible to do? – David542 Sep 10 '14 at 15:50
  • You need to have read access to the source object to copy it. If it's not owned by you, doesn't have permissions specifically allowing you to read it, and isn't public, you can't copy it. –  Sep 10 '14 at 15:55
0

It is possible to use pre-signed URLs to PUT objects directly to S3. I don't believe you could get it work with POST because POST has to encode the body of the request but perhaps there is a way to do that, too. This SO thread should be useful:

How to generate a temporary url to upload file to Amazon S3 with boto library?

Community
  • 1
  • 1
garnaat
  • 44,310
  • 7
  • 123
  • 103
  • Right, but isn't that generating a temp file from a file you already have access to? – David542 Sep 10 '14 at 15:50
  • It is, indeed. Sorry, I didn't read your question carefully enough. In that case, the answer is no, unfortunately. See http://stackoverflow.com/questions/7586533/is-it-possible-to-upload-to-s3-directly-from-url-using-post?rq=1 for more discussion. – garnaat Sep 10 '14 at 15:53