How do I import functions from a module in pyspark

Asked Dec 10 '20 at 07:08

Active Dec 10 '20 at 07:08

Viewed 304 times

I use a spark-submit cmd to run a pyspark script. I have a module with all my utilities. I need to be able to import functions from this module, but I'm getting "module not found error". I have zipped my module and uploaded it using --archive in spark submit. I tried to use zipimport to extract the module from the zip but I'm still unable to read the module. How do I load functions from my module into my pyspark script?

importer = zipimport.zipimporter("module.zip")
mod = importer.load_module("model")

How do I access functions from "mod" object? Any alternatives to zipimport will also be highly appreciated

asked Dec 10 '20 at 07:08

Harsha Reddy

1

You can try the method described here : https://stackoverflow.com/a/39779271/3238085 – user238607 Dec 10 '20 at 12:42
1

Thanks. passing it as --py files works. No need to even use zipimport in this case. – Harsha Reddy Dec 27 '20 at 05:47

How do I import functions from a module in pyspark

0 Answers0