Python way of setting a s3 subdirectory path and deleting it using boto

Question

I have a path like below:

s3://edl-landing/lu/hello2/

Under which I have two tables as shown below:

Now, each table has parquet data in the below folders format:

Now, I want to delete the data for 10th and 11th November. So, below is my python code using the boto client but it doesn't delete the objects from S3. I don't get any errors. Also, I tried various solutions from this link but it also doesn't delete the actual data from S3.

prefix = "lu/hello2/"
s3 = boto3.resource('s3')
bucket = s3.Bucket(name="edl-landing")
FilesNotFound = True
blankList=[]
for obj in bucket.objects.filter(Prefix=prefix):
     #print(obj.key)
     blankList.append(obj.key.split('/')[2])
blankList = set(blankList) // 2 names
while ("" in blankList):
  blankList.remove("")
datesList = ['2020-11-10','2020-11-11']
for i in blankList:
  for j in datesList:
    path = "s3://edl-landing/lu/hello2/"+i+"/edl_load_ts="+j+"/"
    print(path)
    bucket.objects.filter(Prefix=path).delete()
    print("All the objects have been deleted for the mentioned dates...")

Where am I going wrong? I am running it via an EC2 instance.

score 0 · Answer 1 · answered Dec 11 '20 at 14:30

0

You are passing a complete S3 URI to bucket.objects.filter(Prefix=path).

This function requires an S3 object prefix, not an S3 URI.

You should have been able to verify that your code was attempting to delete zero objects because bucket.objects.filter(Prefix=path) returned zero objects.

Pass the relevant prefix: "lu/hello2/"+i+"/edl_load_ts="+j+"/"

answered Dec 11 '20 at 14:30

jarmod

71,565
16
115
122

Okay let me try it. Actually I am new to AWS concepts so wasn't aware about this – whatsinthename Dec 11 '20 at 15:13
No problem. Looks like you got the prefix correct in your earlier call to `bucket.objects.filter()`. – jarmod Dec 11 '20 at 15:14
Yeah the path which I passed is correct. So what should I pass? The path variable or the one which you suggested? – whatsinthename Dec 11 '20 at 15:24
Pass the correct S3 key prefix. It appears to be `"lu/hello2/"+i+"/edl_load_ts="+j+"/"` – jarmod Dec 11 '20 at 15:28

Python way of setting a s3 subdirectory path and deleting it using boto

1 Answers1