Error reading in multiple json files Pyspark

Question

I'm trying to read about 300 single json files into pyspark. I can read a single one, but as soon as I use a wildcard it throughs an error:

IllegalArgumentException: 'Unsupported class file major version 56'

I've tried applying the following code:

import pyspark
from pyspark import SparkContext, SparkConf

conf = SparkConf()
sc = SparkContext(appName='azure_test', conf=conf)
sqlContext = pyspark.SQLContext(sc)

data = sqlContext.read.json('test_1*.json')

I'd expect the output to be a DF of the jsons but instead got the error as mentioned above.

Try the below code text = sc.textFile("file1.json,file2.json") ddff = spark.read.json(text) — dassum, Jul 25 '19 at 14:10
@ dassum that worked but now when I try to use .count() on the df the same error is thrown — Russel Wilcox-Cline, Jul 25 '19 at 15:33
Possible duplicate of [Pyspark error - Unsupported class file major version 55](https://stackoverflow.com/questions/53583199/pyspark-error-unsupported-class-file-major-version-55) — pault, Jul 25 '19 at 17:59

score 0 · Answer 1 · answered Jul 25 '19 at 14:15

from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext

sc = SparkContext("local[2]")
spark = SparkSession.builder.master("local[2]").getOrCreate()

text = sc.textFile("file1,file2...")
ddff = spark.read.json(text)

or put all the files in some folder and use the folder location

sqlContext.read.json("/tmp/test")

score 0 · Answer 2 · answered Jul 25 '19 at 15:46

0

I think there are no issues with your code but Spark not yet compatible with Java-12.

Run with java-8 and then try to read the json files

import pyspark
from pyspark import SparkContext, SparkConf

conf = SparkConf()
sc = SparkContext(appName='azure_test', conf=conf)
sqlContext = pyspark.SQLContext(sc)

data = sqlContext.read.json('test_1*.json')

from Spark-2.0:

spark.read.option("multiline",True).json("<file_path_to_test_1*.json>").show()

answered Jul 25 '19 at 15:46

notNull

30,258
4
35
50

@Russel did it worked? If yes accept the answer to close the thread :) https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work – notNull Jul 29 '19 at 13:51

Error reading in multiple json files Pyspark

2 Answers2