5

I have tried to access the secret {{secrets/secrectScope/Key}} in advanced tab of databricks cluster and it is working fine. But when I try to use the same in databricks init script, it is not working it.

What are the steps to do that?

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
ahmed
  • 273
  • 2
  • 8
  • 16
  • Please post sample code and describe the issue you're having (i.e error message). There is very little information in this question – Nick.Mc Apr 04 '21 at 03:34

2 Answers2

5

Another answer is correct regarding the syntax of the secrets reference (so-called "secret paths"), but it won't work for init scripts, although it will work for Spark code itself.

To pass the secret to the init script you need to put the secrets path into the "Environment Variables" sections of the Spark configuration tab, like this:

enter image description here

And after that you can use the variable by name inside the init script:

if [ -n "$SECRET_VAR" ]; then
  do_something_with_it
fi
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • 1
    And how would you add the environment variable to every new cluster including job clusters? Would you do it via cluster policies `spark_env_vars.`? – Triamus Nov 09 '22 at 16:33
  • 1
    Yes, you should be able to use cluster policies. – Alex Ott Nov 09 '22 at 21:07
  • Do you have a pointer how I can set a secret path as an environment variable in a cluster policy? I couldn't find anything in the docs how to use secrets in cluster policies given the json syntax. – Triamus Nov 11 '22 at 16:38
  • I think I solved it after some trial and error as so `"spark_env_vars.MYVAR": { "type": "fixed", "value": "{{secrets/my_scope/mysecret}}" },` – Triamus Nov 11 '22 at 16:52
1

Here are the steps to access secrets in databricks initscript:

  1. Go to cluster
  2. Click Edit next to the Cluster information.
  3. On the Configure Cluster page, click Advanced Options.
  4. On the Spark tab, enter the following Spark Config:

enter image description here

Sample ini code:

 fs.azure.account.auth.type.chepragen2.dfs.core.windows.net OAuth
 fs.azure.account.oauth.provider.type.chepragen2.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
 fs.azure.account.oauth2.client.id.chepragen2.dfs.core.windows.net {<!-- -->{secrets/KeyVaultName/ClientID}}
 fs.azure.account.oauth2.client.secret.chepragen2.dfs.core.windows.net {<!-- -->{secrets/KeyVaultName/ClientSecret}}
 fs.azure.account.oauth2.client.endpoint.chepragen2.dfs.core.windows.net https://login.microsoftonline.com/<Directory_ID>/oauth2/token

For more details, refer Azure Databricks - configure the cluster to read secrets from the secret scope.

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42