14

I need to find all the files with a specific prefix. For example:

raw/client/Hist/2017/*/*/Tracking_*.zip

I tried this line of code but it does not work:

    import boto3
    client = boto3.client("s3", aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
    client.list_objects(Bucket="myBucket", Prefix="raw/client/Hist/2017/*/*/Tracking_*.zip")
Community
  • 1
  • 1
Eric Bellet
  • 1,732
  • 5
  • 22
  • 40
  • 4
    The [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects) on the `Prefix` parameter explicitly says >Limits the response to keys that begin with the specified prefix. So you'll need to get all keys starting with `raw/client/Hist/2017/` then filter them further yourself. – lee-pai-long Oct 28 '19 at 12:54
  • @Eric Bellet Did you get a solution using prefix wild card . I am trying to figure out – Kar Sep 04 '20 at 21:25
  • Hi, unfortunately no – Eric Bellet Sep 07 '20 at 06:59
  • @Kar See accepted answer if you are still looking for a solution. – gbeaven Jan 21 '21 at 15:04
  • You could consider using fnmatch https://stackoverflow.com/a/11427183 – Shmack Dec 02 '22 at 23:36

1 Answers1

19

You won't be able to do this using boto3 without first selecting a superset of objects and then reducing it further to the subset you need via looping. However, you could use Amazon's data wrangler library and the list_objects method, which supports wildcards, to return a list of the S3 keys you need:

import awswrangler as wr
objects = wr.s3.list_objects('s3://myBucket/raw/client/Hist/2017/*/*/Tracking_*.zip')
gbeaven
  • 1,522
  • 2
  • 20
  • 40