3

I have a python 3.5 notebook in databricks. I have a requirement to execute databricks notebook cells based on some conditions. I didn't see any functionality out of the box.

I have tried creating a python egg with the below code and installed it in databricks cluster.

def skip(line, cell=None):
'''Skips execution of the current line/cell if line evaluates to True.'''
  if eval(line):
    return

  get_ipython().ex(cell)

def load_ipython_extension(shell):

  '''Registers the skip magic when the extension loads.'''
  shell.register_magic_function(skip, 'line_cell')

def unload_ipython_extension(shell):
  '''Unregisters the skip magic when the extension unloads.'''
  del shell.magics_manager.magics['cell']['skip']

But while I'm trying to load that with extension using %load_ext skip_cell it's throwing an error saying "The module is not an IPython module". Any help or suggestion is appreciated. Thanks.

Samrat De
  • 63
  • 1
  • 4

2 Answers2

1

Databricks notebooks are not based on Jupyter/IPython, which is why you are seeing that error.

If you are trying to build conditional workflows I would recommend combining the Notebook Workflows functionality with the Databricks REST API. This will allow you to control the flow of your program based on conditional statements and results of other processes.

Think of a notebook as a function that can be parameterized to accept and return exit values.

For an example, see the official documentation here.

Raphael K
  • 2,265
  • 1
  • 16
  • 23
1

Now there is support to run notebooks based on condition

if <condition>:
    dbutils.notebook.run("notebook-name", 60, {"argument": "data", "argument2": "data2", ...})

More details here https://docs.databricks.com/notebooks/notebook-workflows.html#example

prashanth
  • 4,197
  • 4
  • 25
  • 42
  • the issue here would be that the %run /notebook_path/ does not need any exit command and will get all the objects/functions from the other notebook while the dbutils.notebook.run() doesn't and you explicitly need to specify the exit values – George Sotiropoulos Sep 07 '22 at 11:52
  • @GeorgeSotiropoulos not sure about this. But the run command works very similar to the %run and it works fine without the exit command which is optional. – prashanth Sep 14 '22 at 07:45