SSL Error accessing azure datastore for Azure Auto ML

Question

I am implementing Azure AutoML dashboard in a docker container. When I access container without Docker it works. But in docker it gives SSL Error.

def upload_dataset_to_blob(ws):

    datastore = ws.get_default_datastore()
    datastore.upload_files(files=[
                           '/usr/src/mediafiles/train.csv'], target_path='beeeerrr-dataset/tabular/', overwrite=True, show_progress=True)
    datastore.upload_files(files=[
                           '/usr/src/mediafiles/valid.csv'], target_path='beeeerrr-dataset/tabular/', overwrite=True, show_progress=True)
    datastore.upload_files(files=[
                           '/usr/src/mediafiles/test.csv'], target_path='beeeerrr-dataset/tabular/', overwrite=True, show_progress=True)

    train_dataset = Dataset.Tabular.from_delimited_files(
        validate=False,
        path=[(datastore, 'beeree-dataset/tabular/train.csv')])
    valid_dataset = Dataset.Tabular.from_delimited_files(
        validate=False,
        path=[(datastore, 'beeree-dataset/tabular/valid.csv')])
    test_dataset = Dataset.Tabular.from_delimited_files(

        path=[(datastore, 'beeree-dataset/tabular/test.csv')])

    return train_dataset, valid_dataset, test_dataset

This is the error I am getting

Uploading an estimated of 1 files
app_1     | Uploading /usr/src/mediafiles/train.csv
app_1     | Uploaded /usr/src/mediafiles/train.csv, 1 files out of an estimated total of 1
app_1     | Uploaded 1 files
app_1     | Uploading an estimated of 1 files
app_1     | Uploading /usr/src/mediafiles/valid.csv
app_1     | Uploaded /usr/src/mediafiles/valid.csv, 1 files out of an estimated total of 1
app_1     | Uploaded 1 files
app_1     | Uploading an estimated of 1 files
app_1     | Uploading /usr/src/mediafiles/test.csv
app_1     | Uploaded /usr/src/mediafiles/test.csv, 1 files out of an estimated total of 1
app_1     | Uploaded 1 files
app_1     | <bound method DataReference._get_normalized_path of $AZUREML_DATAREFERENCE_blob_test_data>
app_1     | Internal Server Error: /azureml/train/
app_1     | Traceback (most recent call last):
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/data/dataset_error_handling.py", line 65, in _validate_has_data        
app_1     |     dataflow.verify_has_data()
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/_loggerfactory.py", line 206, in wrapper
app_1     |     return func(*args, **kwargs)
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/dataflow.py", line 875, in verify_has_data
app_1     |     if len(self.take(1)._to_pyrecords()) == 0:
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/dataflow.py", line 792, in _to_pyrecords
app_1     |     self._engine_api.execute_anonymous_activity(
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/_aml_helper.py", line 38, in wrapper
app_1     |     return send_message_func(op_code, message, cancellation_token)
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/api.py", line 120, in execute_anonymous_activity
app_1     |     response = self._message_channel.send_message('Engine.ExecuteActivity', message_args, cancellation_token)
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/engine.py", line 291, in send_message
app_1     |     raise_engine_error(response['error'])
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/dataprep/api/errorhandlers.py", line 10, in raise_engine_error
app_1     |     raise ExecutionError(error_response)
app_1     | azureml.dataprep.api.errorhandlers.ExecutionError:
app_1     | Error Code: ScriptExecution.DatastoreResolution.Unexpected
app_1     | Failed Step: XXXXXXXXXXXXXXXXXXXXXXXXX
app_1     | Error Message: ScriptExecutionException was caused by DatastoreResolutionException.
app_1     |   DatastoreResolutionException was caused by UnexpectedException.
app_1     |     Unexpected failure making request to fetching info for Datastore 'workspaceblobstore' in subscription: 'XXXXXXXXXXXXXXXXXXXXXXXXX', resource group: 'django-env', workspace: 'ml-demo-main'. Using base service url: https://centralus.experiments.azureml.net. HResult: 0x80131501.
app_1     |       The SSL connection could not be established, see inner exception.
app_1     | | session_id=XXXXXXXXXXXXXXXXXXXXXXXXX
app_1     |
app_1     | During handling of the above exception, another exception occurred:
app_1     |
app_1     | Traceback (most recent call last):
app_1     |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
app_1     |     response = get_response(request)
app_1     |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
app_1     |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
app_1     |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
app_1     |     return self.dispatch(request, *args, **kwargs)
app_1     |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
app_1     |     return handler(request, *args, **kwargs)
app_1     |   File "/usr/src/app/azure_ml/views.py", line 50, in get
app_1     |     azureml_train1()
app_1     |   File "/usr/src/app/azure_ml/rough.py", line 39, in azureml_train1
app_1     |     train_dataset, valid_dataset, test_dataset = upload_dataset_to_blob(ws)
app_1     |   File "/usr/src/app/utils/azure_ml/dataset.py", line 28, in upload_dataset_to_blob
app_1     |     train_dataset = Dataset.Tabular.from_delimited_files(
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/data/_loggerfactory.py", line 126, in wrapper
app_1     |     return func(*args, **kwargs)
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/data/dataset_factory.py", line 322, in from_delimited_files
app_1     |     dataflow = _transform_and_validate(
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/data/dataset_factory.py", line 966, in _transform_and_validate
app_1     |     _validate_has_data(dataflow, 'Cannot load any data from the specified path. '
app_1     |   File "/usr/local/lib/python3.8/site-packages/azureml/data/dataset_error_handling.py", line 68, in _validate_has_data
app_1     |     raise DatasetValidationError(error_message + '\n' + e.compliant_message, exception=e)
app_1     | azureml.data.dataset_error_handling.DatasetValidationError: DatasetValidationError:
app_1     |     Message: Cannot load any data from the specified path. Make sure the path is accessible and contains data.
app_1     | ScriptExecutionException was caused by DatastoreResolutionException.
app_1     |   DatastoreResolutionException was caused by UnexpectedException.
app_1     |     Unexpected failure making request to fetching info for Datastore 'workspaceblobstore' in subscription: 'XXXXXXXXXXXXXXXXXXXXXXXXX', resource group: 'django-env', workspace: 'ml-demo-main'. Using base service url: https://centralus.experiments.azureml.net. HResult: 0x80131501.
app_1     |       Failed due to inner exception of type: HttpRequestException
app_1     | | session_id=XXXXXXXXXXXXXXXXXXXXXXXXX
app_1     |     InnerException None
app_1     |     ErrorResponse
app_1     | {
app_1     |     "error": {
app_1     |         "code": "UserError",
app_1     |         "message": "Cannot load any data from the specified path. Make sure the path is accessible and contains data.\nScriptExecutionException was caused by DatastoreResolutionException.\n  DatastoreResolutionException was caused by UnexpectedException.\n    Unexpected failure making request to fetching info for Datastore 'workspaceblobstore' in subscription: 'XXXXXXXXXXXXXXXXXXXXXXXXX', resource group: 'django-env', workspace: 'ml-demo-main'. Using base service url: https://centralus.experiments.azureml.net. HResult: 0x80131501.\n      Failed due to inner exception of type: HttpRequestException\n| session_id=XXXXXXXXXXXXXXXXXXXXXXXXX"
app_1     |     }
app_1     | }

It uploads the file successfully but while accessing back it gives following error

I am using Service Principal for authentication.

def get_workspace():

    svr_pr = ServicePrincipalAuthentication(
        tenant_id=settings.TENANT_ID,
        service_principal_id=settings.SERVICE_PRINCIPAL_ID,
        service_principal_password=settings.SERVICE_PRINCIPAL_PASSWORD
    )

    ws = Workspace(
        subscription_id=settings.SUBSCRIPTION_ID,
        resource_group=settings.RESOURCE_GROUP,
        workspace_name=settings.WORKSPACE_NAME,
        auth=svr_pr
    )

    return ws

Answer 1

I faced the same problem, and after some debugging, I found the following in the log files:

Failed to retrieve datastore with exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. 
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.TypeInitializationException: The type initializer for 'SslMethods' threw an exception.
---> System.TypeInitializationException: The type initializer for 'Ssl' threw an exception.
---> System.TypeInitializationException: The type initializer for 'SslInitializer' threw an exception.
---> Interop+Crypto+OpenSslCryptographicException: error:0E076071:configuration file routines:MODULE_RUN:unknown module name
   at Interop.SslInitializer..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl.SslV2_3Method()
   at Interop.Ssl.SslMethods..cctor()
   --- End of inner exception stack trace ---

This pointed me to this thread: https://github.com/dotnet/runtime/issues/29855

It seems .NET 2.1 tries to use OpenSSL 1.0 while the distro's config at /etc/ssl/openssl.conf is not compatible with that. This has been reported to exist for multiple Debian versions.

Possible solutions:

• Comment out the ssl_conf value in /etc/ssl/openssl.cnf
• Uninstall OpenSSL 1.0.x
• Set CLR_OPENSSL_VERSION_OVERRIDE=1.1 to force OpenSSL 1.1 to be used. (2.1 prefers binding to OpenSSL 1.0.x because that's what it shipped with... it only uses 1.1 if explicitly told to, or if 1.0 isn't available)

I'd tried the last one, and in my case it resolved the problem. I am using Debian Buster.