How do I use databricks-cli without manual configuration

Question

I want to use databricks cli:

databricks clusters list

but this requires a manual step that requires interactive work with the user:

databricks configure --token

Is there a way to use databricks cli without manual intervention so that you can run it as part of a ci/cd pipeline?

score 5 · Answer 1 · answered Nov 15 '19 at 20:19

5

You can just export the variables DATABRICKS_HOST and DATABRICKS_TOKEN. With those variables, you don't need a config file.

answered Nov 15 '19 at 20:19

usingnamespace

93
1
8

score 5 · Answer 2 · edited Sep 09 '22 at 14:30

5

As mentioned above by @usingnamespace, and from the official docs:

CLI 0.8.0 and above supports environment variables, an environment variable setting takes precedence over the setting in the configuration file.

DATABRICKS_HOST
DATABRICKS_USERNAME
DATABRICKS_PASSWORD
DATABRICKS_TOKEN

With that, not only you will not be exposing sensitive data in clear text files (~/.databrickscfg), you won't need to add any more code to your script.

edited Sep 09 '22 at 14:30

Marco Roy

4,004
7
34
50

answered Jan 31 '20 at 09:33

Gee VB

183
3
8

This should be the accepted answer now. It's much better than populating a config file. – Marco Roy Sep 09 '22 at 14:37

score 3 · Accepted Answer · answered Aug 14 '18 at 08:34

The following bash script, configured the databricks cli automatically:

echo "configuring databrick-cli authentication"

declare DATABRICKS_URL="https://westeurope.azuredatabricks.net"
declare DATABRICKS_ACCESS_TOKEN="authentication_token_generated_from_databricks_ux"

declare dbconfig=$(<~/.databrickscfg)
if [[ $dbconfig = *"host = "* && $dbconfig = *"token = "* ]]; then
  echo "file [~/.databrickscfg] is already configured"
else
  if [[ -z "$DATABRICKS_URL" || -z "$DATABRICKS_ACCESS_TOKEN" ]]; then
    echo "file [~/.databrickscfg] is not configured, but [DATABRICKS_URL],[DATABRICKS_ACCESS_TOKEN] env vars are not set"
  else
    echo "populating [~/.databrickscfg]"
    > ~/.databrickscfg
    echo "[DEFAULT]" >> ~/.databrickscfg
    echo "host = $DATABRICKS_URL" >> ~/.databrickscfg
    echo "token = $DATABRICKS_ACCESS_TOKEN" >> ~/.databrickscfg
    echo "" >> ~/.databrickscfg
  fi
fi

apparently you can now log on with a username, and password... but on Azure i can never get it to work #voodoo — m1nkeh, Apr 24 '19 at 12:33
I am trying to run this in R. Do you know how should I go about it using this bash script to run in R? — 89_Simple, Sep 10 '19 at 10:13

score 2 · Answer 4 · answered Apr 09 '20 at 15:17

2

This is a PowerShell version of the script.

write-host Configure databricks access
$Env:DATABRICKS_CONFIG_FILE = "$(System.DefaultWorkingDirectory)/.databrickscfg"
Set-Location $(System.DefaultWorkingDirectory)
Set-Content .databrickscfg "[DEFAULT]"
Add-Content .databrickscfg "host = https://westeurope.azuredatabricks.net/"
Add-Content .databrickscfg "token = $(db-token)"

answered Apr 09 '20 at 15:17

hui chen

1,004
14
19

To make the configure effective, the databricks-cli commands must be called in the same PS session. – hui chen Apr 12 '20 at 08:53

How do I use databricks-cli without manual configuration

4 Answers4