Multiline JSON is not parsing

Question

Error on console
scala> var njson = spark.read.option("multiLine", "true").json("C:/Users/Onkar/Desktop/new.json") njson: org.apache.spark.sql.DataFrame = [_corrupt_record: string]

My JSON:

[{
"Name": "Onkar",
"Age": 24,
"Place": "Baramati"
}]

{ "Name": "Onkar", "Age": 24, "Place": "Baramati", "Number": 7709526425 } — Onkar Jadhav, May 21 '20 at 06:50
Are you saying that your file contains only `{ "Name": "Onkar", "Age": 24, "Place": "Baramati", "Number": 7709526425 }` and the code above didn't manage to load the data frame? — ernest_k, May 21 '20 at 07:00
As explained in the first comment, spark json reader expects record (doesn't matter if multi-line or not) to be JSON record, not array - because it treats each key as a column - can't be done with array. What did you expect to be the content of your dataframe after the read? — Yaroslav Fyodorov, May 21 '20 at 07:03

Srinivas · Accepted Answer · 2020-05-21T11:35:54.010

0

In spark version 2.2.0 onwards parsing array of json object is supported. Lower version of spark parsing array of json object is not supported or not parsing properly.

Please check below code.

scala> spark.version
res47: String = 2.2.0

scala> "cat /tmp/app.json".!
[{
"Name": "Onkar",
"Age": 24,
"Place": "Baramati",
"Number": 7709526425
}]
res48: Int = 0

scala> val jdf = spark.read.option("multiLine","true").json("/tmp/app.json")
jdf: org.apache.spark.sql.DataFrame = [Age: bigint, Name: string ... 2 more fields]

scala> jdf.printSchema
root
 |-- Age: long (nullable = true)
 |-- Name: string (nullable = true)
 |-- Number: long (nullable = true)
 |-- Place: string (nullable = true)


scala> jdf.show(false)
+---+-----+----------+--------+
|Age|Name |Number    |Place   |
+---+-----+----------+--------+
|24 |Onkar|7709526425|Baramati|
+---+-----+----------+--------+

spark version 2.1.1 has some issue as its not parsing array of json object & This issue has been fixed in newer versions.

scala> spark.version
res2: String = 2.1.1

scala> val jdf = spark.read.option("multiLine","true").json("/tmp/app.json")
jdf: org.apache.spark.sql.DataFrame = [_corrupt_record: string]

scala> jdf.printSchema
root
 |-- _corrupt_record: string (nullable = true)

edited May 21 '20 at 11:35

answered May 21 '20 at 07:46

Srinivas

8,957
2
12
26

my version spark is 2.1.1 and scala 2.11.8 is there any problem with this ? – Onkar Jadhav May 21 '20 at 10:59
val jdf = spark.read.json(sc.wholeTextFiles("C:/Users/Onkar/Desktop/new.json").values) This worked for me .. – Onkar Jadhav May 21 '20 at 11:13
spark 2.1.1 does not support .option() – Onkar Jadhav May 21 '20 at 11:16
I have updated answer, Please accept or upvote if this solution helps to solve your problem. – Srinivas May 21 '20 at 11:28
Parsing array of json object in spark 2.1.0 or 2.1.1 will not work & same issue has fixed in spark 2.2.0 version... why down vote ??? – Srinivas May 21 '20 at 11:37
Spark 2.1.1 will support option().. that why are json is able to read only thing is not parsing – Srinivas May 21 '20 at 12:13
Thank U for ur help .. and I am new here .. I think by mistake I down voted u .. sorry for that .. – Onkar Jadhav May 21 '20 at 12:35
No problem..Thank you.. – Srinivas May 21 '20 at 12:40

Multiline JSON is not parsing

1 Answers1