0

I want to create a python notebook on my desktop that pass an input to another notebook in databricks, and then return the output of the databricks notebook. For example, my local python file will pass a string into a databricks notebook, which will reverse the string and then output the result back to my local python file. What would be the best way to achieve this?

This is what I tried but when I try to create a new run, I get this error. Is my json formatted incorrectly or am I missing something else? Thanks

import os
from databricks_cli.sdk.api_client import ApiClient
from databricks_cli.clusters.api import ClusterApi

os.environ['DATABRICKS_HOST'] = "https://adb-################.##.azuredatabricks.net/"
os.environ['DATABRICKS_TOKEN'] = "token-value" 

api_client = ApiClient(host=os.getenv('DATABRICKS_HOST'), token=os.getenv('DATABRICKS_TOKEN'))

runJson = """
        {
        "name": "test job",
        "max_concurrent_runs": 1,
        "tasks": [
            {
            "task_key": "test",
            "description": "test",
            "notebook_task":
                {
                "notebook_path": "/Users/user@domain.com/api_test"
                },
            "existing_cluster_id": "cluster_name",
            "timeout_seconds": 3600,
            "max_retries": 3,
            "retry_on_timeout": true
            }
            ]
        }
        """

runs_api = RunsApi(api_client)
runs_api.submit_run(runJson)

Error: Response from server:

{ 
  'error_code': 'MALFORMED_REQUEST',
  'message': 'Invalid JSON given in the body of the request - expected a map'}
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
J. Doe
  • 165
  • 5
  • 16

1 Answers1

0

You should provide the payload as Python dict, not as string. Just remove """ around the runJson payload.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • thanks that worked. i got everything else to work except the last part of returning the output. do i need to create another post or can i just add it to this one? – J. Doe May 12 '23 at 15:26
  • Better to create a separate one – Alex Ott May 12 '23 at 15:29
  • done. https://stackoverflow.com/questions/76239157/how-to-call-databricks-notebook-from-python-with-rest-api – J. Doe May 13 '23 at 07:32