11

I'm trying to create a new cluster in Databricks on Azure using databricks-cli.

I'm using the following command:

databricks clusters create --json '{ "cluster_name": "template2", "spark_version": "4.1.x-scala2.11" }'

And getting back this error:

Error: {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required field: size"}

I can't find documentation on this issue, would be happy to receive some help.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Mor Shemesh
  • 2,689
  • 1
  • 24
  • 36

3 Answers3

15

I found the right answer here.

The correct format to run this command on azure is:

databricks clusters create --json '{ "cluster_name": "my-cluster", "spark_version": "4.1.x-scala2.11", "node_type_id": "Standard_DS3_v2", "autoscale" : { "min_workers": 2, "max_workers": 50 } }'
Mor Shemesh
  • 2,689
  • 1
  • 24
  • 36
  • Yes however this creates Multi mode cluster. If you need to create single node cluster different set of parameters you need to pass – JemHah Oct 25 '22 at 13:44
  • I used the API and it worked like a charm, I used it to clone an existing cluster and just had to rename and remove the cluster_id. Btw, your link is not working anymore. https://docs.databricks.com/api/workspace/clusters/create – Henrique Brisola Jul 21 '23 at 13:54
4

Just to add to the answer that @MorShemesh gave, you can also use a path to a JSON file instead of specifying the JSON at the command line.

databricks clusters create --json-file /path/to/my/cluster_config.json

If you are managing lots of clusters this might be an easier approach.

Raphael K
  • 2,265
  • 1
  • 16
  • 23
0
databricks clusters create --json "{ "cluster_name": "custpm-cluster", "spark_version": "4.1.x-scala2.09", "node_type_id": "Standard_DS3_v2", "autoscale" : { "min_workers": 2, "max_workers": 50 }}"
Yogesh
  • 1
  • 3