5

With databricks-connect we can successfully run codes written in Databricks or Databricks notebook from many IDE. Databricks has also created many magic commands to support their feature with regards to running multi-language support in each cell by adding commands like %sql or %md. One issue I am facing currently is when I try to execute Databricks notebooks in Pycharm is as follows:

  • How to execute Databricks specific magic command from PyCharm. E.g. Importing a script or notebook in Done in Databricks using this command-

    %run
'./FILE_TO_IMPORT'

Where as in IDE from FILE_TO_IMPORT import XYZ works.

Again everytime I download Databricks notebook it comments out the magic commands and that makes it impossible to be used anywhere outside Databricks environment.

It's really inefficient to convert all databricks magic command everytime I want to do any developement.

Is there any configuration I could set which automatically detects Databricks specific magic commands?

Any solution to this will be helpful. Thanks in Advance!!!

Rohit Mishra
  • 441
  • 4
  • 17

1 Answers1

3

Unfortunately, as per the databricks-connect version 6.2.0-

" We cannot use magic command outside the databricks environment directly. This will either require creating custom functions but again that will only work for Jupyter not PyCharm"

Again, since importing py files requires %run magic command so this also becomes a major issue. A solution to this is by converting the set of files to be imported as a python package and add it to the cluster via Databricks UI and then import and use it in PyCharm. But this is a very tedious process.

Rohit Mishra
  • 441
  • 4
  • 17
  • you can implement the last part without the tedium by writing a script to create the package and upload it to dbfs (using dbfs cli) allowing you to install that updated package every time you run your notebook by just running the script. – Chappy Hickens Dec 12 '22 at 18:50
  • can you provide a source for the quoted text please? – Matteo De Felice Jan 16 '23 at 14:52