Unable to read json file from dbfs which inturn the data is getting corrupted in python spark

Asked Jun 11 '19 at 10:11

Active Jun 11 '19 at 10:38

Viewed 640 times

I am unable to read json files from dbfs which is getting corrupted dataframe.

I have tried

dfx = spark.read.option("multiline","true").json("/FileStore/tables/vv.json")

dfx = spark.read.option("multiline", "true").json("/FileStore/tables/vv.json")

output

dfx:pyspark.sql.dataframe.DataFrame = [_corrupt_record: string]

edited Jun 11 '19 at 10:38

Steven

14,048
6
38
73

asked Jun 11 '19 at 10:11

vikas madoori

Please verify whether your json file is proper or corrupt. This error arises when you have corrupt json record. – Prathik Kini Jun 11 '19 at 10:18
yes json file is proper only but dataframe is getting corrupted @Imprathik could you please help me – vikas madoori Jun 11 '19 at 10:34
it is working in local perfectly but not in databricks – vikas madoori Jun 11 '19 at 10:35
encoding issue? – Steven Jun 11 '19 at 10:39
Have you checked this https://stackoverflow.com/questions/38895057/reading-json-with-apache-spark-corrupt-record – Prathik Kini Jun 11 '19 at 10:44

Unable to read json file from dbfs which inturn the data is getting corrupted in python spark

0 Answers0