Pass JVM arguments in Databricks Jobs API

Question

I want to pass JVM arguments to a REST request using Jobs API in Databricks. Is it possible to do it? Any hints would help.

Something like -Dconfig-file=app.conf to the spark job.

Answer 1

You can use "spark_conf" attribute in the REST API Jobs .

spark_conf: An object containing a set of optional, user-specified Spark configuration key-value pairs. You can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

Example: Spark confs: {"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or {"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}

How to use in Jobs API: To create a cluster enabled for table access control, specify the following spark_conf property in your request body:

curl -X POST https://<databricks-instance>/api/2.0/clusters/create -d'
{
  "cluster_name": "my-cluster",
  "spark_version": "5.2.x-scala2.11",
  "node_type_id": "Standard_DS3_v2",
  "spark_conf": {
    "spark.databricks.acl.dfAclsEnabled":true,
    "spark.databricks.repl.allowedLanguages": "python,sql"
  },
  "num_workers": 1,
  "custom_tags":{
     "costcenter":"Tags",
     "applicationname":"Tags1"
  }
}'

For more details, refer "API examples" and "How to add spark configuration".