Azure ML: Upload File to Step Run's Output - Authentication Error

Question

During a PythonScriptStep in an Azure ML Pipeline, I'm saving a model as joblib pickle dump to a directory in a Blob Container in the Azure Blob Storage which I've created during the setup of the Azure ML Workspace. Afterwards I'm trying to upload this model file to the step run's output directory using

Run.upload_file (name, path_or_stream)

(for the function's documentation, see https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.core.run(class)?view=azure-ml-py#upload-file-name--path-or-stream--datastore-name-none-)

Some time ago when I created the script using the azureml-sdk version 1.18.0, everything worked fine. Now, I've updated the script's functionalities and upgraded the azureml-sdk to version 1.33.0 during the process and the upload function now runs into the following error:

Traceback (most recent call last):
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_file_utils/upload.py", line 64, in upload_blob_from_stream
    validate_content=True)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 93, in execute_func_with_reset
    return ClientBase._execute_func_internal(backoff, retries, module_logger, func, reset_func, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 367, in _execute_func_internal
    left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 399, in _handle_retry
    raise error
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 358, in _execute_func_internal
    response = func(*args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/blockblobservice.py", line 614, in create_blob_from_stream
    initialization_vector=iv
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 98, in _upload_blob_chunks
    range_ids = [f.result() for f in futures]
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 98, in <listcomp>
    range_ids = [f.result() for f in futures]
  File "/opt/miniconda/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/opt/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/opt/miniconda/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 210, in process_chunk
    return self._upload_chunk_with_progress(chunk_offset, chunk_bytes)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 224, in _upload_chunk_with_progress
    range_id = self._upload_chunk(chunk_offset, chunk_data)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 269, in _upload_chunk
    timeout=self.timeout,
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/blockblobservice.py", line 1013, in _put_block
    self._perform_request(request)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/storageclient.py", line 432, in _perform_request
    raise ex
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/storageclient.py", line 357, in _perform_request
    raise ex
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/storageclient.py", line 343, in _perform_request
    HTTPError(response.status, response.message, response.headers, response.body))
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/_error.py", line 115, in _http_error_handler
    raise ex
azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5d4e1b7e-c01e-0070-0d47-9bf8a0000000
Time:2021-08-27T13:30:02.2685991Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcw
2021-08-27T13:19:56Z
2021-08-28T13:29:56Z
/blob/mystorage/azureml/ExperimentRun/dcid.98d11a7b-2aac-4bc0-bd64-bb4d72e0e0be/outputs/models/Model.pkl

2019-07-07
b

</AuthenticationErrorDetail></Error>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/jobs/.../azureml-setup/context_manager_injector.py", line 243, in execute_with_context
    runpy.run_path(sys.argv[0], globals(), run_name="__main__")
  File "/opt/miniconda/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/opt/miniconda/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/opt/miniconda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 318, in <module>
    main()
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 286, in main
    path_or_stream=model_path)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 53, in wrapped
    return func(self, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 1989, in upload_file
    datastore_name=datastore_name)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 114, in upload_artifact
    return self.upload_artifact_from_path(artifact, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 107, in upload_artifact_from_path
    return self.upload_artifact_from_stream(stream, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 99, in upload_artifact_from_stream
    content_type=content_type, session=session)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 88, in upload_stream_to_existing_artifact
    timeout=TIMEOUT, backoff=BACKOFF_START, retries=RETRY_LIMIT)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_file_utils/upload.py", line 71, in upload_blob_from_stream
    raise AzureMLException._with_error(azureml_error, inner_exception=e)
azureml._common.exceptions.AzureMLException: AzureMLException:
    Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
    StorageAccount: mystorage
    ContainerName: azureml
    StatusCode: 403
    InnerException Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5d4e1b7e-c01e-0070-0d47-9bf8a0000000
Time:2021-08-27T13:30:02.2685991Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcw
2021-08-27T13:19:56Z
2021-08-28T13:29:56Z
/blob/mystorage/azureml/ExperimentRun/dcid.98d11a7b-2aac-4bc0-bd64-bb4d72e0e0be/outputs/models/Model.pkl

2019-07-07
b

</AuthenticationErrorDetail></Error>
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: mystorage\n\tContainerName: azureml\n\tStatusCode: 403",
        "inner_error": {
            "code": "Auth",
            "inner_error": {
                "code": "Authorization"
            }
        }
    }
}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 318, in <module>
    main()
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 286, in main
    path_or_stream=model_path)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 53, in wrapped
    return func(self, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 1989, in upload_file
    datastore_name=datastore_name)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 114, in upload_artifact
    return self.upload_artifact_from_path(artifact, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 107, in upload_artifact_from_path
    return self.upload_artifact_from_stream(stream, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 99, in upload_artifact_from_stream
    content_type=content_type, session=session)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 88, in upload_stream_to_existing_artifact
    timeout=TIMEOUT, backoff=BACKOFF_START, retries=RETRY_LIMIT)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_file_utils/upload.py", line 71, in upload_blob_from_stream
    raise AzureMLException._with_error(azureml_error, inner_exception=e)
UserScriptException: UserScriptException:
    Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
    StorageAccount: mystorage
    ContainerName: azureml
    StatusCode: 403
    InnerException AzureMLException:
    Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
    StorageAccount: mystorage
    ContainerName: azureml
    StatusCode: 403
    InnerException Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5d4e1b7e-c01e-0070-0d47-9bf8a0000000
Time:2021-08-27T13:30:02.2685991Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcw
2021-08-27T13:19:56Z
2021-08-28T13:29:56Z
/blob/mystorage/azureml/ExperimentRun/dcid.98d11a7b-2aac-4bc0-bd64-bb4d72e0e0be/outputs/models/Model.pkl

2019-07-07
b

</AuthenticationErrorDetail></Error>
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: verovisionstorage\n\tContainerName: azureml\n\tStatusCode: 403",
        "inner_error": {
            "code": "Auth",
            "inner_error": {
                "code": "Authorization"
            }
        }
    }
}
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: mystorage\n\tContainerName: azureml\n\tStatusCode: 403"
    }
}

As far as I can tell from the code of the azureml.core.Run class and the subsequent function calls, the Run object tries to upload the file to the step run's output directory using SAS-Token-Authentication (which fails). This documentation article is linked in the code (but I don't know if this relates to the issue): https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas#service-sas-example

Did anybody encounter this error as well and knows what causes it or how it can be resolved?

Best, Jonas

Can you share how you create the `Run` object? – Anders Swanson Aug 27 '21 at 15:00 — Anders Swanson, Aug 27 '21 at 15:00

score 1 · Answer 1 · answered Aug 27 '21 at 15:03

1

We’ve seen the before, it’s annoying. I think the answer is to go to the data stores page of the AML Studio UI and manually enter the storage account key again.

answered Aug 27 '21 at 15:03

Anders Swanson

3,637
1
18
43

Thank you for the fast reply! Sadly this approach didn't solve the problem :-/ Do you have any other ideas? The Run-Object is created in the AML Pipeline PythonScriptStep through ` from azureml.core import Run RUN_OBJECT = Run.get_context() ` – Jonas Aug 30 '21 at 11:57
1

During further contemplation, I saw that I can set the Authentication Type of the Datastore in Azure ML to "Account key" (the default setting) or "SAS token". Given that the SAS Token Authentication fails, I thought about changing the setting to "SAS token". But, as this would be the only case where I (or rather the azureml-sdk) use the SAS token, and therefore this might break the authentication at other places in the script, I guess. What do you think? – Jonas Aug 30 '21 at 12:07
I empathize with your experience! definitely not ideal. I'd start with trying the both choices to see if you can anything to work. That'll help us understand what's going on. Afterwards, we can decide what's best for your scenario. Does "Account key" work for you? – Anders Swanson Aug 30 '21 at 21:51
also, if you or your organization have Azure priority support, I strongly recommend opening a support ticket for this. I'll also let some Azure ML PMs know about your issue, as I am confused as well. – Anders Swanson Aug 30 '21 at 21:52
1

Can you please provide a feedback from the Datastore page in the Azure Machine Learning Portal UI using the smiley icon next to question mark. Also, do include your email in the feedback so that we can follow-up on the feedback to get more information. Thanks. – ynpandey Aug 31 '21 at 00:17
Hi @ynpandey , due to the recently released v1.36 of the azureml-sdk, I decided to give it another shot to check if the bug has been fixed already. Sadly the error still occurs. Do you have any information on when you plan to fix this bug? Is there anything else I could do in the meantime to work around this issue? Best, Jonas – Jonas Nov 15 '21 at 12:24

Azure ML: Upload File to Step Run's Output - Authentication Error

1 Answers1