3

I want to use dbutils command for accessing secrets in my pyspark job submitting through Spark-Submit inside Jobs on Databricks. While using dbutils command it is giving error dbutils not defined.

Is there is the way to use dbutils in a pyspark job other than a notebook?

Tried the following solutions:

1) import DBUtils, according to this solution. But this is not Databricks dbutils.

2) import pyspark.dbutils import DBUtils , according to this solution. But this also didn't work.

pyspark job as shown in below diagram: image

Dheerajkumar Solanki
  • 31
  • 5

1 Answers1

-1

What I understand by your question- How do I access the secret value from the key-vault in Databricks notebook?
So to access the keys you have to link your Databricks notebook to the key-vault. This linking has to be done by using the Scopes.
In the below url you have to provide the details:

https://westeurope.azuredatabricks.net/#secrets/createScope

If you open the link then you will land to a secret scope creation page: Scope creation Page Link may be vary depends on the area/region of subscription. In my case it is "westeurope".

You have to provide 3 values on the page:
1) Scope Name
2) DNS Name
3) Resource ID

Scope name have to be define manually and rest of the two values has to copy from the key-vault.
Now go to your Azure key-vault and click on properties section.
KeyVaultScreenShot

Copy the DNS Name and Resource ID values and paste it into required cells of the Scope creation page.

That's it. Now your notebook will point to your key vault.

I have given az-kv-scope-test name to my scope.

Lets assume my key-vault having a secret with the name db_password and I want to access it in my notebook.

So to access the secret value from the key vault inside the Databricks notebook I have to use the below command.

    // variable declaration and initialization.    
    val my_kv_scope = "az-kv-scope-test"    
    val password = dbutils.secrets.get(scope = my_kv_scope, key = "db_password")

Following link will guide you to get more information on secrete scopes.
https://docs.databricks.com/security/secrets/index.html

I hope this will give you some clarity on the Key-vault values accessibility in notebook.

venus
  • 1,188
  • 9
  • 18