Azure Storage: Blob: Python: Get indicator if there are blobs at all

Question

I have a Python app. In this context I want to retrieve the blob references from an Azure Storage container that match a certain prefix and then delete all the blobs in one go. I tried the following:

container_client: ContainerClient = ContainerClient.from_connection_string(conn_str=storage_account_connection_string, container_name=container_name)

blob_list: ItemPaged[BlobProperties] = container_client.list_blobs(name_starts_with=prefix)

container_client.delete_blobs(*blob_list, delete_snapshots="include")

This works fine as long as there are blobs that match the prefix. But if that is not the case I get an exception when trying to execute delete_blobs:

tuple index out of range

I don't want to work with try except and I also don't want to iterate first. I would like to have an indicator that tells me if there are blobs at all without the need to do extra calls.

How can I do that?

Thanks

EDIT: Based on what has been suggested by @Gaurav the following approach works:

from azure.storage.blob import ContainerClient, BlobProperties
from azure.core.paging import ItemPaged
from typing import List

blob_paged: ItemPaged[BlobProperties] = container_client.list_blobs(name_starts_with=prefix)
blob_list: List[dict] = list(blob_paged)
number_of_blobs: int = len(blob_list)

if number_of_blobs > 0:
    container_client.delete_blobs(*blob_list, delete_snapshots="include")
    log.debug(f"Deleted '{ number_of_blobs }' blobs and snapshots...")   
else:
    log.debug(f"No blobs to be deleted...")

Three things you should be aware about:

Using list() will resolve the iterator and load all blobs into memory
blob_paged can't be used anymore as argument for delete_blobs after being resolved
When using blob_list as argument for delete_blobs it will log a warning like Failed to parse headers... (Bug?). The blobs still get deleted.

Gaurav Mantri · Accepted Answer · 2020-07-29T06:45:34.057

3

delete_blobs method makes use of Blob Batch operation to delete multiple blobs in a single request. According to the documentation, maximum number of items in a batch can be 256 or the maximum payload size is 4MB (Ref: https://learn.microsoft.com/en-us/rest/api/storageservices/blob-batch#remarks).

I believe you're getting this error is because you're either sending more than 256 blobs in your delete_blobs method or the payload is more than 4MB in size.

UPDATE

You will also get the error if the items in the blobs_list are zero. You can use the following code to see the number of items (Ref: Getting number of elements in an iterator in Python):

number_of_blobs = len(list(blobs_list))

edited Jul 29 '20 at 06:45

answered Jul 28 '20 at 16:18

Gaurav Mantri

128,066
12
206
241

Thanks for your response. This exception occurs (reproducible) when there are no blobs that match the prefix. – quervernetzt Jul 28 '20 at 17:04
Oh, I forgot to mention that the service will also return an error if the batch size is zero. – Gaurav Mantri Jul 28 '20 at 17:09
Yes, but the question is how to determine that the batch size is zero respectively that there are no blobs to be retrieved? Is there no property that indicates that? Thanks – quervernetzt Jul 28 '20 at 19:10
Can't you simply check the length or count of `blob_list` variable to determine that? – Gaurav Mantri Jul 28 '20 at 23:54
Oh! That’s surprising. Let me try it out. I’ll revert shortly. – Gaurav Mantri Jul 29 '20 at 04:01
As you can see it is not a list but ItemPaged[BlobProperties] and does not include the actual blobs but only an iterator to retrieve them lazily (as far as I know). – quervernetzt Jul 29 '20 at 04:08
Updated my answer. HTH. – Gaurav Mantri Jul 29 '20 at 06:45
Thanks, this seem to work. Please see my edits above. – quervernetzt Jul 29 '20 at 08:16

Azure Storage: Blob: Python: Get indicator if there are blobs at all

1 Answers1