How to get the whole cluster information in azure databricks at the runtime?

Question

The below code was working for the older version and the version has changed the code is not working in databricks.

Latest Version :12.0 (includes Apache Spark 3.3.1, Scala 2.12)

dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags()

what is the alteranative for this code?

Error:py4j.security.Py4JSecurityException: Method public scala.collection.immutable.Map com.databricks.backend.common.rpc.CommandContext.tags() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext

What older version is the above code working fine for? – Saideep Arikontham Feb 16 '23 at 09:52 — Saideep Arikontham, Feb 16 '23 at 09:52

Kombajn zbożowy · Answer 1 · 2023-02-17T08:01:52.760

2

You can get most of cluster info directly from Spark config:

%scala
val p = "spark.databricks.clusterUsageTags."
spark.conf.getAll
  .collect{ case (k, v) if k.startsWith(p) => s"${k.replace(p, "")}: $v" }
  .toList.sorted.foreach(println)

%python
p = "spark.databricks.clusterUsageTags."
conf = [f"{k.replace(p, '')}: {v}" for k, v in spark.sparkContext.getConf().getAll() if k.startswith(p)]
for l in sorted(conf): print(l)

[...]
clusterId: 0123-456789-0abcde1
clusterLastActivityTime: 1676449848620
clusterName: test
clusterNodeType: Standard_F4s_v2
[...]

edited Feb 17 '23 at 08:01

answered Feb 16 '23 at 10:41

Kombajn zbożowy

8,755
3
28
60

My Admin provided access to use only python.Do you have any python code for that? – Venkatesh Feb 16 '23 at 11:27
Can be also done in Python, I added in my answer. – Kombajn zbożowy Feb 17 '23 at 08:02

score 0 · Answer 2 · answered Feb 16 '23 at 10:29

0

I have created the following cluster with DBS runtime of 12.0 as shown below:

enter image description here

The above command has given the output as required without any error:

enter image description here

But if it is the cluster information that you need, then you can use Clusters 2.0 API. The following code would work:



import requests
import json

my_json = {"cluster_id": spark.conf.get("spark.databricks.clusterUsageTags.clusterId")}    

auth = {"Authorization": "Bearer <your_access_token>"}

response = requests.get('https://<workspace_url>/api/2.0/clusters/get', json = my_json, headers=auth).json()
print(response)

enter image description here

answered Feb 16 '23 at 10:29

Saideep Arikontham

5,558
2
3
11

print(dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags()) this command is throwing an error.Could you please check – Venkatesh Feb 16 '23 at 11:19
i dont have access token is there any alteranative approach to get the whole cluster information at run time – Venkatesh Feb 16 '23 at 11:20
The print is working fine as well (didn't include image as there is a lot of sensitive information). Try the solution given by @Kombajn zbożowy if you do not have access token. – Saideep Arikontham Feb 16 '23 at 11:24
do we need to add any configuration ,for me still i am facing the issue – Venkatesh Feb 16 '23 at 11:26
Not Sure as I'm not at all facing the error. What cluster details do you need for your requirement? – Saideep Arikontham Feb 16 '23 at 11:27
Here is the cluster details that i am using Databricks Run time version:12.0 (includes Apache Spark 3.3.1, Scala 2.12) – Venkatesh Feb 16 '23 at 11:30
I meant what details are you trying to get? – Saideep Arikontham Feb 16 '23 at 11:30
At run time which user is executing ? which language the code was written? notebook path? clusterid? clustername? api_url? notebook_path? so at run time we need this info – Venkatesh Feb 16 '23 at 11:32
How to find the databricks notebookid do you have any command? – Venkatesh Feb 16 '23 at 14:14
As far as I know, there is not way to programmatically get the notebook id – Saideep Arikontham Mar 07 '23 at 09:21

How to get the whole cluster information in azure databricks at the runtime?

2 Answers2