26

I have build the Spark-csv and able to use the same from pyspark shell using the following command

bin/spark-shell --packages com.databricks:spark-csv_2.10:1.0.3

error getting 

>>> df_cat.save("k.csv","com.databricks.spark.csv")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/pyspark/sql/dataframe.py", line 209, in save
    self._jdf.save(source, jmode, joptions)
  File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError

Where should I place the jar file in my spark pre-built setup so that I will be able to access spark-csv from python editor directly as well.

Abhishek Choudhary
  • 8,255
  • 19
  • 69
  • 128

6 Answers6

26

At the time I used spark-csv, I also had to download commons-csv jar (not sure it is still relevant). Both jars where in the spark distribution folder.

  1. I downloaded the jars as follow:

    wget http://search.maven.org/remotecontent?filepath=org/apache/commons/commons-csv/1.1/commons-csv-1.1.jar -O commons-csv-1.1.jar<br/>    
wget http://search.maven.org/remotecontent?filepath=com/databricks/spark-csv_2.10/1.0.0/spark-csv_2.10-1.0.0.jar -O spark-csv_2.10-1.0.0.jar

  2. then started the python spark shell with the arguments:

    ./bin/pyspark --jars "spark-csv_2.10-1.0.0.jar,commons-csv-1.1.jar"

  3. and read a spark dataframe from a csv file:

    from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
df = sqlContext.load(source="com.databricks.spark.csv", path = "/path/to/you/file.csv")
df.show()
m0nhawk
  • 22,980
  • 9
  • 45
  • 73
Yannick Marcon
  • 276
  • 3
  • 3
23

Another option is to add the following to your spark-defaults.conf:

spark.jars.packages com.databricks:spark-csv_2.11:1.2.0
kentt
  • 543
  • 5
  • 12
17

Instead of placing the jars in any specific folder a simple fix would be to start the pyspark shell with the following arguments:

bin/pyspark --packages com.databricks:spark-csv_2.10:1.0.3

This will automatically load the required spark-csv jars.

Then do the following to read the csv file:

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv')
df.show()
d18s
  • 53
  • 8
Jimmy
  • 2,165
  • 1
  • 17
  • 13
  • 2
    How would I modify your answer when I'm setting the configuration parameters from IPython using `SparkConf().set("key", "value")` before creating the Spark Context? I don't see a "packages" key, and setting "spark.jars" to a path containing the downloaded jar didn't work. – dnlbrky Jul 02 '15 at 19:49
  • is there a way to do what @dnlbrky wants? – Paul Jan 08 '16 at 20:03
  • If spark context already created, you probably can't change most of it parameters. – Tagar Feb 22 '16 at 06:59
  • 4
    SparkConf().set("spark.jars.packages","com.databricks:sparkcsv_2.10:1.0.3").. I know m too late to respond @dnlbrky – Jimmy Nov 18 '16 at 11:55
  • @Jimmy - Thanks. This worked for me, `SparkSession.builder.config('spark.jars.packages','com.databricks:sparkc‌​sv_2.10:1.0.3' ).getOrCreate()`. @dnlbrky @Paul – Clay Dec 07 '17 at 21:46
  • UNRESOLVED DEPENDENCIES , Thats what im getting – yunus Mar 28 '19 at 11:03
4

Assuming the session/context hasn't been created yet:

import os

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.databricks:spark-csv_2.10:1.3.0 pyspark-shell'
Kamil Sindi
  • 21,782
  • 19
  • 96
  • 120
  • 1
    What if I want to add multiple packages? I am trying `--packages com.databricks:spark-avro_2.11:4.0.0 databricks:spark-deep-learning:1.1.0-spark2.3-s_2.11 pyspark-shell` but I got `Java gateway process exited before sending its port number` – argenisleon Aug 27 '18 at 16:44
0

Below command helped me -: With Scala 2.10 version

/opt/mapr/spark/spark-1.5.2/bin/spark-shell --master local[*] --packages com.databricks:spark-csv_2.10:1.4.0

Has below dependencies -:

com.databricks#spark-csv_2.10;1.4.0!spark-csv_2.10.jar (2043ms)
org.apache.commons#commons-csv;1.1!commons-csv.jar (419ms)
com.univocity#univocity-parsers;1.5.1!univocity-parsers.jar (1481ms)
user3815062
  • 11
  • 4
0

first find out the path of the spark. for example for pyspark

    which pyspark

it will return you the path for example like this- /home/ubuntu/bin/pyspark

then run this command by change the path as per your spark path general-: path --packages com.databricks:spark-csv_2.10:1.0.3

    /home/ubuntu/bin/pyspark --packages com.databricks:spark-csv_2.10:1.0.3
Shubham Sinha
  • 25
  • 2