0

I have created a pyspark project and wrapped up in a .whl file which I am then using as a package to instantiate a interactive Pyspark Shell.

<PYTHONPATH>/pyspark --py-files sample-project.whl --name pyspark-test --jars anotherSupportedLibrary.jar

Once it has initialized I would like to use this project modules and files to execute certain actions. But I am unable to import certain utility methods from the project files.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.4.502
      /_/

Using Python version 3.7.13 (default, Mar 29 2022 02:18:16)
SparkSession available as 'spark'.
>>> from module1.spark_entry import DriverClass
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'module1'
>>> 

Appreciate any leads or inputs on this.

0 Answers0