Azure Databricks: How to add Spark configuration in Databricks cluster

Question

I am using a Spark Databricks cluster and want to add a customized Spark configuration.
There is a Databricks documentation on this but I am not getting any clue how and what changes I should make. Can someone pls share the example to configure the Databricks cluster.
Is there any way to see the default configuration for Spark in the Databricks cluster.

Answer 1

1. You can set cluster config in the compute section in your Databricks workspace. Go to compute (and select cluster) > configuration > advanced options:

2. Or, you can set configs via a notebook.

   %python spark.conf.set("spark.sql.name-of-property", value)

Answer 2

You have many ways to set up the default cluster configs:

1. Manually in the "compute" tab (as mentioned before): Go to Compute > Select a cluster > Advanced Options > Spark

2. Via notebook (as mentioned before): In a cell of your databricks notebook, you can set any spark configuration for that session/job by running the "spark.conf.set" command like spark.conf.set("spark.executor.memory","4g")

3. Using JOB CLI API: If you are aiming to deploy jobs programmatically in a multi-environment fashion (e.g. Dev, Staging, Production):

Useful links!

• https://docs.databricks.com/api/workspace/jobs/create
• https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs-2.0-api
• https://spark.apache.org/docs/latest/configuration.html