databricks cli to update job shcedule

Question

I have configured the Databricks cli locally and able to connect to the Azure Databricks cluster. Link reference used for my workstation - git

Below command list the jobs successfully with the id

$ databricks jobs list --profile dev

Say if I wanted to update only the schedule (cron expression) to a specific job which is already deployed in workspace, i don't see any option to do it using databricks CLI.

Note: In my case the the jobs are created using job definition json, is used to create the jobs in the cluster. This json doesn't have the schedule info to start with.

Is there are any options available to update only schedule, after the job is created or deployed in the workspace?

There is an option to run the command immediately, databricks jobs run-now.

The REST API configuration https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate

score 3 · Answer 1 · answered Jun 01 '22 at 07:44

3

To update the job use the databricks jobs reset command.

Databricks Azure jobs CLI docs

Documentation claims that partial updates are possible, but whenever I try to only update the schedule it complains about parameters missing.

A way around it is to read the job settings first and then editing the job json before updating:

databricks jobs get --job-id 1234 > my-job.json

then update the my-job.json and use it with reset command:

databricks jobs reset --job-id 1234 --json-file my-job.json

answered Jun 01 '22 at 07:44

botchniaque

4,698
3
35
63

I noticed the same "parameters missing" issue when I tried to use the `reset` command. I have created an issue on databricks cli repo, please upvote - https://github.com/databricks/databricks-cli/issues/606#issuecomment-1446074865 – Georgi Koemdzhiev Mar 07 '23 at 12:24

score -1 · Accepted Answer · answered Dec 14 '21 at 03:38

The Databricks jobs CLI supports calls to two versions of the Databricks Jobs REST API: versions 2.1 and 2.0. Version 2.1 adds support for orchestration of jobs with multiple tasks; see Jobs with multiple tasks and Jobs API updates.

Going through the MS docs I see, you can use the update request to change an existing job.

databricks jobs update --job-id 246 --json-file update-job.json

schedule information block in the input json file.

"schedule": {
      "quartz_cron_expression": "0 0 0 * * ?",
      "timezone_id": "US/Pacific",
      "pause_status": "UNPAUSED"
    }

Refer: MS DOC - An example JSON document representing a single-task format job for API 2.0:

{
  "job_id": 27,
  "settings": {
    "name": "Example notebook",
    "existing_cluster_id": "1201-my-cluster",
    "libraries": [
      {
        "jar": "dbfs:/FileStore/jars/spark_examples.jar"
      }
    ],
    "email_notifications": {},
    "timeout_seconds": 0,
    "schedule": {
      "quartz_cron_expression": "0 0 0 * * ?",
      "timezone_id": "US/Pacific",
      "pause_status": "UNPAUSED"
    },
    "notebook_task": {
      "notebook_path": "/notebooks/example-notebook",
      "revision_timestamp": 0
    },
    "max_concurrent_runs": 1,
    "format": "SINGLE_TASK"
  },
  "created_time": 1504128821443,
  "creator_user_name": "user@databricks.com"
}

Ok. Using JSON definition the job schedule can be updated. Was looking for option like $ databricks jobs update --job-id 246 --schedule '0 0 0 ...'. — Tim, Dec 14 '21 at 04:52
Datbricks jobs cli does not have the option `update`. use `databricks jobs reset` to update jobs. — botchniaque, Jun 01 '22 at 07:45

databricks cli to update job shcedule

2 Answers2