1

Is it possible to refer to a databricks secret in my property name like this:

fs.azure.account.auth.type.{{secrets/my_scope/my_secret1}}.dfs.core.windows.net OAuth
fs.azure.account.auth.type.{{secrets/my_scope/my_secret2}}.dfs.core.windows.net OAuth

or is secrets only allowed for the assigned value?

My databricks workflows fails with the current message Failure to initialize configuration for storage account [REDACTED].dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

I am not 100% certain it is a problem with the configuration, so I just want to make sure it is possible before looking for other issues

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
x-platformcoder
  • 13
  • 3

2 Answers2

0

I think it might be better to ask this question on the Databricks community forum, as it is very product specific and not a general programming question. Also I'd recommend reading the documentation on secret redaction. I think the substituion is possible like that, since it showing as [REDACTED] in the printed notebook output is an indication that it was inserted correctly.

RvdV
  • 406
  • 3
  • 10
0

No, it's not possible - secrets are matched by the full value, not substituted inside the string - it's described in the documentation. It's also easy to check. For example, if you start a cluster with such value, and then use Scala snippet to filter out necessary values:

%scala

spark.conf.getAll.filter { _._1.contains(".dfs.core.windows.net")}

then you should get something like this:

res1: scala.collection.immutable.Map[String,String] = Map(
  fs.azure.account.auth.type.{{secrets/my_scope/my_secret1}}.dfs.core.windows.net -> OAuth
)

showing that no substitution happened. The error message is a bit misleading because it's detected that you refers the secret and redacted it.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132