-1

So I am writing a few scripts, one of those scripts is a helper script. When I make changes to the helper script (make sure the changes are saved) I cannot see those changes through the other script. For example I changed the value of a variable in the helper script and I want to use that updated value in the other script, when I run the other script I can not see the change. I have to restart the cluster to see the change.

Is there a reason why I cannot see instant changes in databricks like we can see in local machines?

harsh panday
  • 83
  • 1
  • 10
  • Please provide more details other than generic terms like "script" (are they Python Files being imported, Notebooks, Python Wheels, JARs, do you use Git Repos in Databricks, ??). Sharing actual code would be helpful to understand how you are "using a script from another script". – Zach King Jul 18 '23 at 23:36
  • Yes, they are python files. I am writing code in Databricks itself. Basically, I have 2 python files one of which contains a bunch if helper functions which I use in the other file to do some analysis. But when I update something in the helper file I can not use that update instantly in the other file. I need to restart the cluster to use that update. The code is not the problem, it's just basic python code, the file synchronization is the issue. – harsh panday Jul 19 '23 at 14:27

1 Answers1

1

When using Python files (modules) -- and not to be confused with Python notebooks -- the module is interpreted and loaded upon import. Once you have imported a module, changes to the module are not reflected unless the Python interpreter is restarted, or you can do it dynamically at runtime using the importlib built-in module, as detailed in the Answer here: How do I unload (reload) a Python module?

from importlib import reload  # Python 3.4+
import my_module

# Do some things
print(my_module.x) 

# Change things in `my_module`

# Reload the module at runtime
my_module = reload(my_module)

# Do things again; changes are reflected
print(my_module.x)
Zach King
  • 798
  • 1
  • 8
  • 21