Bulk Generate Pre-Signed URLs boto3

Question

I am currently using the following to create a pre-signed url for a bucket resource:

bucket_name = ...
key = ...
s3_client = ...

s3_client.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": bucket_name,
        "Key": key
    },
    ExpiresIn=100
)

This works fine. However, I was wondering if it was possible to generate pre-signed urls for multiple keys in one request? Or is it required to make one request for each key? I didn't find anything useful in the docs regarding this topic. I'm looking for something like this:

bucket_name = ...
keys = [...]
s3_client = ...

# Returns an array of pre-signed urls, in the same order as `keys`
s3_client.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": bucket_name,
        "Keys": keys
    },
    ExpiresIn=100
)

score 34 · Accepted Answer · answered Dec 27 '18 at 23:51

34

Generating presigned URLs is actually done locally, without requiring a call to AWS. This is because all necessary information (Bucket, Key, Secret Key) is known locally and can generate the signature.

Therefore, feel free to call that function repeatedly since there is no network/service overhead.

In general, there should be no need to bulk-generate URLs. Instead, whenever your application wishes to reference a resource (eg an image on an HTML page), it can quickly call the generate_presigned_url() function with an appropriate timeout.

answered Dec 27 '18 at 23:51

John Rotenstein

241,921
22
380
470

9

Note that if generating multiple signed URLs in a loop, you'll probably want to reuse the S3 client object. Instantiating that object is likely to be unnecessarily time consuming/resource intensive. – Michael - sqlbot Dec 28 '18 at 02:58
Brilliant; I was wondering why their was no bulk option. Thanks! – treyhakanson Dec 28 '18 at 03:42
Really appreciate your answer sir, I have a query that what If I want to generate presigned url for multiple files in a bucket. For example: currently for 1.jpg I am generating a single presinged url and using that ur I can upload content to my bucket perfectly but what if I want to upload 4 files. e.g. 1.jpg, 2.jpg, 3.jpg, 4.jpg – A l w a y s S u n n y Jul 26 '21 at 07:02
@AlwaysSunny See: [Pre-signed url for multiple files?](https://stackoverflow.com/a/67830706/174777) – John Rotenstein Jul 26 '21 at 07:04
@JohnRotenstein thanks for the link, it works to generate presigned url for multiple files and works to post single file/object at a time, is there any way so I can upload all the files at once using that presigned url i generate for them? just curious to know :) thanks in advance – A l w a y s S u n n y Jul 26 '21 at 09:43
I guess programatically with AWS POST it is not possible to upload multiple files at once, but aws cli has something like `aws s3 sync local_folder s3://bucket_name/` that does the trick. please correct me If I am wrong – A l w a y s S u n n y Jul 26 '21 at 09:47
@AlwaysSunny It is not possible to upload multiple files in one request. The AWS CLI runs multiple threads and uploads files via multiple parallel API calls. – John Rotenstein Jul 26 '21 at 10:34
how can the secret key be known locally? If it's available locally, it's no longer "secret" – Dmitry B. Nov 14 '21 at 06:15
@DmitryB. I'm not sure what you mean by "locally". A Secret key is effectively a password. Your application can use it to generate a pre-signed URL. The signature on that URL is only valid for that particular object, within an expiry period and it uses a hash so it cannot be reverse-engineered. But, yes, the app needs access to the Secret Key. – John Rotenstein Nov 14 '21 at 06:33
Actuallly, by "local" I meant the same thing as you. After made my comment, though, I did some more reading and I realized that by "secret key" you probably meant "secret access key" of a local profile (as in access credentials). What I still don't quite understand (need more digging) is how AWS can efficiently map a URL signature to the principal that signed it (when it validates the URL upon access). – Dmitry B. Nov 15 '21 at 03:47
A pre-signed URL contains the Access Key in clear text (effectively a username), plus a 'signature', which is a hash of the permissions and the Secret Access Key. So, it can lookup the Secret Access Key and created its own hash to verify the signature. – John Rotenstein Nov 15 '21 at 04:00

score 1 · Answer 2 · answered Jan 11 '22 at 18:11

As @John Rotenstein mentioned in his response, you can repeatedly call this function inside a For Loop.

Here is an example:

def create_presigned_urls(s3Client, bucket_name: str, key: str, expires_in: int):
  """Create presigned_urls
  Args:
      s3Client (s3 Class): boto3 S3 Class
      bucket_name
      key
      expires_in: The number of seconds the presigned URL is valid for.

  Returns:
      (string): presigned URL
  """
  presigned_url = s3Client.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": bucket_name,
        "Key": key
    },
    ExpiresIn=expires_in
  )
  return presigned_url

For loop:

s3Client = boto3.client('s3')
bucket_name = 'BUCKET_NAME'
expires_in = 3600
list_file_url = []

for index, unit in df.iterrows():
    key = df['key_name']
    url = create_presigned_urls(s3Client, bucket_name, key, expires_in)
    list_file_url.append(url)

Bulk Generate Pre-Signed URLs boto3

2 Answers2

Linked