3

I am a beginner in Azure Databricks and I want to use APIs to create cluster and submit job in python. I am stuck as I am unable to do so. Also if I have an existing cluster how will the code look like? I got job id after running this code but unable to see any output.

import requests

DOMAIN = ''
TOKEN = ''
response = requests.post(
  'https://%s/api/2.0/jobs/create' % (DOMAIN),
  headers={'Authorization': 'Bearer %s' % TOKEN},
  json={
       "name": "SparkPi spark-submit job",
       "new_cluster": {
       "spark_version": "7.3.x-scala2.12",
       "node_type_id": "Standard_DS3_v2",
       "num_workers": 2
       },
       "spark_submit_task": {
       "parameters": [
           "--class",
            "org.apache.spark.examples.SparkPi",
            "dbfs:/FileStore/sparkpi_assembly_0_1.jar",
            "10"
            ]
        }
    }
)
if response.status_code == 200:
  print(response.json())
else:
  print("Error launching cluster: %s: %s" % (response.json()["error_code"], response.json()["message"]))
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Coderash
  • 47
  • 1
  • 9

1 Answers1

3

Jobs at Databricks could be executed two ways (see docs):

  • on a new cluster - that's how you do it right now
  • on existing cluster - remove the new_cluster block, and add the existing_cluster_id field with the ID of existing cluster. If you don't have a cluster yet, then you can create it via Cluster API

When you create a job, then you get back the job ID that could be used to edit the job or delete it. You can also launch the job using the Run Now API. But if you just want to execute the job without create the Job in the UI, then you need to look onto Run Submit API. Either of the APIs will return the ID of specific job run, and then you can use Run Get API to get status of the job, or Run Get Output API to get the results of execution.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • I am unable to get the output of value of pi, though I am able to see it in logs, can you help me with that if I want to print the value of Pi? – Coderash Jul 15 '21 at 11:49
  • you need to use "Run Get Output API" to get it – Alex Ott Jul 15 '21 at 12:08
  • in my pipeline, I would like to wait until job is completed and then move to next task , how it can be done ? https://stackoverflow.com/questions/73291921/wait-until-databrick-job-is-completed-in-devops-pipeline – alex3465 Aug 09 '22 at 13:05