Azure Python download storage blob returns 'The condition specified using HTTP conditional header(s) is not met.'

Question

With the following code:


import os
from azure.identity import (
    ClientSecretCredential
)

# Import the client object from the Azure library
from azure.storage.blob import BlobClient

t_id = "<tenant_id>"
c_id = "<client_id>"
s_acct = "<storage_account_name>"
s_acct_url = "%s.blob.core.windows.net" % s_acct
sek = "<client_sekret>"

print("+ Setup credentials.")
credential = ClientSecretCredential(t_id, c_id, sek)
print("+ Setup Blob Client")
bobc = BlobClient(s_acct_url, <container_name>, <blob_name>,
                  credential=credential)
print("+ Setup streamer")
ssd = bobc.download_blob()
print("+ Get properties")
print(ssd.get_blob_properties())

But I get the following error:

$ python azdown.py
+ Setting up stream
+
+ Download stream:
+    Size: 136365212160
Traceback (most recent call last):
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 192, in _download_chunk
    _, response = self.client.download(
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_generated\operations\_blob_operations.py", line 179, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The condition specified using HTTP conditional header(s) is not met.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "azdown.py", line 29, in <module>
    download_stream.download_to_stream(my_blob)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 579, in download_to_stream
    self.readinto(stream)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 561, in readinto
    downloader.process_chunk(chunk)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 125, in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 201, in _download_chunk
    process_storage_error(error)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_shared\response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ResourceModifiedError: The condition specified using HTTP conditional header(s) is not met.
RequestId:<request id>
Time:2021-01-10T01:23:24.8981731Z
ErrorCode:ConditionNotMet
Error:None

I looked at [1], but I am using download_to_stream(). Then I tried [2] but my storage is a gen1 container. There was a link that it had something to do with the container's network permissions but the container is set to allow all to download. The problem is, it does download up to a certain point (atm, 1.2G out of 120G) and then it chokes. So it isn't a permission issue but some sort of race condition.

[1] - https://social.msdn.microsoft.com/Forums/azure/en-US/3b4df832-3340-4415-8d93-d71662e1c540/azure-blob-the-condition-specified-using-http-conditional-headers-is-not-met

[2] - https://forums.databricks.com/questions/20415/intermittent-http-error-when-loading-files-from-ad.html

[edit]

I also tried using the connection_string method:

credential = DefaultAzureCredential()

    blob_url = "DefaultEndpointsProtocol=https;AccountName=<storage_acct>;" + \
               "AccountKey=<account_key>;" + \
               "EndpointSuffix=core.windows.net"

    # Create the client object using the storage URL and the credential
    blob_client = BlobClient.from_connection_string(
        blob_url,
        container_name=<container_name>,
        blob_name=<blob_item>)

    if download_now:
        with open(<blob_item>, "wb") as my_blob:
            print("+ Setting up stream")
            download_stream = blob_client.download_blob()
            print("+")
            print("+ Download stream:")
            print("+    Size: %d" % download_stream.size)
            download_stream.download_to_stream(my_blob)

    else:
        print(blob_client.get_blob_properties())

I get the same kind of error even after it has streamed a gig or so.

Error:

Traceback (most recent call last):
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 192, in _download_chunk
    _, response = self.client.download(
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_generated\operations\_blob_operations.py", line 179, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The condition specified using HTTP conditional header(s) is not met.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "azuredl.py", line 52, in <module>
    download_stream.download_to_stream(my_blob)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 579, in download_to_stream
    self.readinto(stream)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 561, in readinto
    downloader.process_chunk(chunk)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 125, in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 201, in _download_chunk
    process_storage_error(error)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_shared\response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ResourceModifiedError: The condition specified using HTTP conditional header(s) is not met.
RequestId:<request id>
Time:2021-01-10T07:43:24.4927426Z
ErrorCode:ConditionNotMet
Error:None

Might anyone have any ideas?

score 1 · Accepted Answer · answered Jan 12 '21 at 04:58

I can repro this issue when I make some updates(such as change content or adding some metadata) on the file that I am downloading:

This issue is due to once the blob gets updated, its Etag changes at the same time. Which causes this issue. Details see this doc.

To solve this issue, you can require a Lise for this blob to add a lock so that this blob will not be edit or delete while downloading.

Try code below:

blob_client = BlobClient.from_connection_string(conn_str='', container_name='',blob_name='')

#require lease that never expires
lease = blob_client.acquire_lease(lease_duration=-1)

with open("<some path>", "wb") as my_blob:
            print("+ Setting up stream")
            download_stream = blob_client.download_blob()
            print("+")
            print("+ Download stream:")
            print("+    Size: %d" % download_stream.size)
            download_stream.download_to_stream(my_blob)

#break lease after download
lease.break_lease()

Thanks! I think that's exactly what is screwing up. I hadn't realized (or remembered) I had something updating that blob. Tricky issue here. — ewokx, Jan 12 '21 at 05:10

Azure Python download storage blob returns 'The condition specified using HTTP conditional header(s) is not met.'

1 Answers1

Linked