Adding a jar file to pyspark after context is created

Question

I am using pyspark from a notebook and I do not handle the creation of the SparkSession. I need to load a jar containing some functions I would like to use while processing my rdds. This is something which you can easily do using --jars which I cannot do in my particular case. Is there a way to access the spark scala context and call the addJar method? I tried to use the JavaGateway (sparksession._jvm...) but have not been successful so far. Any idea?

Thanks Guillaume

score 3 · Answer 1 · answered Mar 23 '17 at 05:44

3

sparksession._jsc.addJar does the job.

answered Mar 23 '17 at 05:44

tog

887
1
12
22

... but it solves my problem only partially as the method is node available on my driver node! – tog Mar 24 '17 at 05:05
I found this very useful [post] (http://stackoverflow.com/questions/37132559/add-jars-to-a-spark-job-spark-submit) – tog Mar 24 '17 at 05:07

score 2 · Answer 2 · answered Sep 13 '17 at 13:28

2

You can try this method, which will add the file to the context of all nodes:

spark.sparkContext.addFile("filename")

answered Sep 13 '17 at 13:28

dirceusemighini

1,344
2
16
35

score 1 · Answer 3 · answered Dec 28 '20 at 10:15

1

distribute xxx.jar by addJar and import it by extraClassPath

spark = SparkSession.builder.config('spark.driver.extraClassPath', 'xxx.jar').getOrCreate()
spark.sparkContext._jsc.addJar('/xxx/xxx/xxx.jar')

answered Dec 28 '20 at 10:15

qq945801210

11
1

Adding a jar file to pyspark after context is created

3 Answers3

Linked