Can't read avro in Jupyter notebook

Question

I already have a SparkContext created and a Spark global variable. When I read ORC files, I can read them as simple as spark.read.format("orc").load("filepath") however, for avro I can't seem to do the same even though I try to import the jar like so:

    spark.conf.set("spark.jars.packages",
    "file:///projects/apps/lib/spark-avro_2.11-3.2.0.jar")

Error:

and then try to read the avro file. I get an error like so: 
Py4JJavaError: An error occurred while calling o65.load.
: org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;

Have you check this link : https://stackoverflow.com/questions/46434255/how-to-query-datasets-in-avro-format — Starfight, Apr 25 '19 at 13:57
@Starfight The link is for spark-shell, I need it for Jupyter notebook — user2896120, Apr 25 '19 at 13:59
I see, but a contributor recommand to use `com.databricks.spark.avro` instead of `avro`, maybe interesting ? — Starfight, Apr 25 '19 at 14:07

score 2 · Accepted Answer · answered Apr 25 '19 at 14:39

2

spark.jars.packages takes Gradle compatible coordinates:

spark.jars.packages  org.apache.spark:spark-avro_2.12:2.4.2

Additionally, as explained in How to load jar dependenices in IPython Notebook, it has to be set before JVM and SparkSession / SparkContext are initialized.

So you have to:

Fix the settings.
Provide these as a configuration or environment variable, before JVM is initialized.

answered Apr 25 '19 at 14:39

user11411285

36
1

Is it possible to destroy the current spark context that was initialized and set the spark context with the jar within jupyter? – user2896120 Apr 25 '19 at 15:12

Can't read avro in Jupyter notebook

1 Answers1