Add Jar to standalone pyspark

Question

I'm launching a pyspark program:

$ export SPARK_HOME=
$ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip
$ python

And the py code:

from pyspark import SparkContext, SparkConf

SparkConf().setAppName("Example").setMaster("local[2]")
sc = SparkContext(conf=conf)

How do I add jar dependencies such as the Databricks csv jar? Using the command line, I can add the package like this:

$ pyspark/spark-submit --packages com.databricks:spark-csv_2.10:1.3.0 

But I'm not using any of these. The program is part of a larger workflow that is not using spark-submit I should be able to run my ./foo.py program and it should just work.

• I know you can set the spark properties for extraClassPath but you have to copy JAR files to each node?
• Tried conf.set("spark.jars", "jar1,jar2") that didn't work too with a py4j CNF exception

Answer 1

2021-01-19 Updated

There are many approaches here (setting ENV vars, adding to $SPARK_HOME/conf/spark-defaults.conf, etc...) other answers already cover these. I wanted to add an answer for those specifically wanting to do this from within a Python Script or Jupyter Notebook.

When you create the Spark session you can add a .config() that pulls in the specific Jar file (in my case I wanted the Kafka package loaded):

spark = SparkSession.builder.appName('my_awesome')\
    .config('spark.jars.packages', 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1')\
    .getOrCreate()

Using this line of code I didn't need to do anything else (no ENVs or conf file changes).

• Note 1: The JAR file will dynamically download, you don't need to manually download it.
• Note 2: Make sure the versions match what you want, so in the example above my Spark version is 3.0.1 so I have :3.0.1 at the end.

Answer 2

Any dependencies can be passed using spark.jars.packages (setting spark.jars should work as well) property in the $SPARK_HOME/conf/spark-defaults.conf. It should be a comma separated list of coordinates.

And packages or classpath properties have to be set before JVM is started and this happens during SparkConf initialization. It means that SparkConf.set method cannot be used here.

Alternative approach is to set PYSPARK_SUBMIT_ARGS environment variable before SparkConf object is initialized:

import os
from pyspark import SparkConf

SUBMIT_ARGS = "--packages com.databricks:spark-csv_2.11:1.2.0 pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS

conf = SparkConf()
sc = SparkContext(conf=conf)

Answer 3

I encountered a similar issue for a different jar ("MongoDB Connector for Spark", mongo-spark-connector), but the big caveat was that I installed Spark via pyspark in conda (conda install pyspark). Therefore, all the assistance for Spark-specific answers weren't exactly helpful. For those of you installing with conda, here is the process that I cobbled together:

1) Find where your pyspark/jars are located. Mine were in this path: ~/anaconda2/pkgs/pyspark-2.3.0-py27_0/lib/python2.7/site-packages/pyspark/jars.

2) Download the jar file into the path found in step 1, from this location.

3) Now you should be able to run something like this (code taken from MongoDB official tutorial, using Briford Wylie's answer above):

from pyspark.sql import SparkSession

my_spark = SparkSession \
    .builder \
    .appName("myApp") \
    .config("spark.mongodb.input.uri", "mongodb://127.0.0.1:27017/spark.test_pyspark_mbd_conn") \
    .config("spark.mongodb.output.uri", "mongodb://127.0.0.1:27017/spark.test_pyspark_mbd_conn") \
    .config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.11:2.2.2') \
    .getOrCreate()

Disclaimers:

1) I don't know if this answer is the right place/SO question to put this; please advise of a better place and I will move it.

2) If you think I have errored or have improvements to the process above, please comment and I will revise.

Answer 4

Finally found the answer after a multiple tries. The answer is specific to using spark-csv jar. Create a folder in you hard drive say D:\Spark\spark_jars. Place the following jars there:

1. spark-csv_2.10-1.4.0.jar (this is the version I am using)
2. commons-csv-1.1.jar
3. univocity-parsers-1.5.1.jar

2 and 3 are dependencies required by spark-csv, hence those two files need to be downloaded too. Go to your conf directory where you have downloaded Spark. In the spark-defaults.conf file add the line:

spark.driver.extraClassPath D:/Spark/spark_jars/*

The asterisk should include all the jars. Now run Python, create SparkContext, SQLContext as you normally would. Now you should be able to use spark-csv as

sqlContext.read.format('com.databricks.spark.csv').\
options(header='true', inferschema='true').\
load('foobar.csv')

Answer 5

From working with PySpark, PostgreSQL, and Apache Sedona, I learned to solve this with 2 methods.

---

Method 1: Download the JAR file and add to spark.jars

In order to use PostgreSQL on Spark, I needed to add the JDBC driver (JAR file) to PySpark.

First, I created a jars directory in the same level as my program and store the postgresql-42.5.0.jar file there.

Then, I simply add this config to SparkSession with: SparkSession.builder.config("spark.jars", "{JAR_FILE_PATH}")

spark = (
    SparkSession.builder
    .config("spark.jars", "jars/postgresql-42.5.0.jar")
    .master("local[*]")
    .appName("Example - Add a JAR file")
    .getOrCreate()
)

---

Method 2: Use Maven Central coordinate and spark.jars.packages

If your dependency JAR files are available on Maven, you can use this method and not have to maintain any JAR file.

Steps

1. Find your package on Maven Central Repository Search

2. Select the correct package artifact and copy the Maven Central coordinate

3. In Python, call SparkSession.builder.config("spark.jars.packages", "{MAVEN_CENTRAL_COORDINATE}").

   spark = (
       SparkSession.builder
       .appName('Example - adding many Maven packages')
   
       .config("spark.serializer", KryoSerializer.getName)
       .config("spark.kryo.registrator", SedonaKryoRegistrator.getName)
       .config("spark.jars.packages",
               "org.postgresql:postgresql:42.5.0,"
               + "org.apache.sedona:sedona-python-adapter-3.0_2.12:1.2.1-incubating,"
               + "org.datasyslab:geotools-wrapper:1.1.0-25.2")
   
       .getOrCreate()
    )
   

Pros of using sparks.jars.packages

• You can add several packages
• You don't have to manage the fat JAR files

Cons of using sparks.jars.packages

The .config("sparks.jars.packages", ...) accept a single parameter, so in order to add several packages, you need to concatenate the package coordinates using , as the delimiter.

"org.postgresql:postgresql:42.5.0,"
+ "org.apache.sedona:sedona-python-adapter-3.0_2.12:1.2.1-incubating,"
+ "org.datasyslab:geotools-wrapper:1.1.0-25.2"

*** The string will not tolerate next line, spaces, or tabs in your code and it will cause nasty bugs that gives out irrelevant error logs.

Answer 6

import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "/python")
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.10.4-src.zip'))

Here it comes....

sys.path.insert(0, <PATH TO YOUR JAR>)

Then...

import pyspark
import numpy as np

from pyspark import SparkContext

sc = SparkContext("local[1]")
.
.
.

Answer 7

For sparkoperator in yml manifest you can use in sparkConf "spark.jars.packages" for severals packages

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: test
  namespace: default
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  sparkVersion: "3.3.2"
  sparkConf:
    "spark.jars.packages": "org.apache.hadoop:hadoop-aws:3.3.2,com.amazonaws:aws-java-sdk-bundle:1.12.99"