How to delete files that matches a specific pattern in S3 bucket?

Question

I have an S3 bucket that I'm saving CSV files for loading them into Redshift. I'm using Python and Boto3 for this. After loading them into Redshift I want to delete specific files that match a pattern that contains the processing ID for my code.

I'm saving my files into S3 bucket as follows

Redshift{processingID}-table1.csv
Redshift{processingID}-table2.csv
Redshift{processingID}-table3.csv
Redshift{processingID}-table4.csv

After processing those files that contains specific ID, I want to delete the processed files from my S3 bucket. How do I specify the pattern.

This is the pattern that I'm trying to delete the files from bucket.

Redshift11-*.csv. Here 11 is the processingID. How do I delete all the files that matches the pattern using boto3?

I've come across this. https://stackoverflow.com/a/53836093/4626254

But it seems like it is searching for the folder as prefix and not the exact pattern of the file.

What's the pattern? is it always located under the same folder? — Amit Baranes, Oct 20 '19 at 19:15
Yes. The files will always be in root directory in the bucket. — Underoos, Oct 20 '19 at 19:16
than the Prefix trick you pointed should work. `Prefix (string) -- Limits the response to keys that begin with the specified prefix.` Since your files always under the root folder, this should work. give it a try and let us know. — Amit Baranes, Oct 20 '19 at 19:18

score 4 · Answer 1 · answered Oct 20 '19 at 19:22

You can do prefix filtering server-side but you'll have to do suffix-filtering client-side. For example:

import boto3
s3 = boto3.resource('s3')

bucket = s3.Bucket('mybucket')
files = [os.key for os in bucket.objects.filter(Prefix="myfolder/Redshift11-")]
csv_files = [file for file in files if file.endswith('.csv')]

print(f'All files: {files}')
print(f'CSV files: {csv_files}')

pyjamaboy · Answer 2 · 2019-10-20T19:34:10.070

There's no way to tell S3 to delete files that meet a specific pattern - you just have to delete one file at a time. You can list keys with a specific prefix

ex: Redshift or application_name_used_as_prefix) by modifying your file naming to have a unique prefix.

Or if you need to rely on regex, then you must specify start and end rules like:

import re

pattern = r"Redshift([0-9]+)-(\w+).csv$"
re.match(pattern, 'Redshift2-table1.csv')

Hope this helps!

How to delete files that matches a specific pattern in S3 bucket?

2 Answers2