I have created a pyspark project and wrapped up in a .whl file which I am then using as a package to instantiate a interactive Pyspark Shell.
<PYTHONPATH>/pyspark --py-files sample-project.whl --name pyspark-test --jars anotherSupportedLibrary.jar
Once it has initialized I would like to use this project modules and files to execute certain actions. But I am unable to import certain utility methods from the project files.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.4.502
/_/
Using Python version 3.7.13 (default, Mar 29 2022 02:18:16)
SparkSession available as 'spark'.
>>> from module1.spark_entry import DriverClass
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'module1'
>>>
Appreciate any leads or inputs on this.