0

I want to convert my nested json into csv ,i used

df.write.format("com.databricks.spark.csv").option("header", "true").save("mydata.csv")

But it can use to normal json but not nested json. Anyway that I can convert my nested json to csv?help will be appreciated,Thanks!

fin
  • 1
  • 1
  • 3
  • When you try to run your program, can you see "UnsupportedOperationException: CSV data source does not support struct..." in your logs? – Carlos Vilchez Sep 22 '16 at 08:37
  • yes i got error " CSV data source does not support struct..." @CarlosVilchez – fin Sep 22 '16 at 08:44

2 Answers2

1

When you ask Spark to convert a JSON structure to a CSV, Spark can only map the first level of the JSON. This happens because of the simplicity of the CSV files. It is just asigning a value to a name. That is why {"name1":"value1", "name2":"value2"...} can be represented as a CSV with this structure: name1,name2, ... value1,value2,... In your case, you are converting a JSON with several levels, so Spark exception is saying that it cannot figure out how to convert such a complex structure into a CSV.

If you try to add only a second level to your JSON, it will work, but be careful. It will remove the names of the second level to include only the values in an array.

You can have a look at this link to see the example for json datasets. It includes an example.

As I have no information about the nature of the data, I can't say much more about it. But if you need to write the information as a CSV you will need to simplify the structure of your data.

Carlos Vilchez
  • 2,774
  • 28
  • 30
0

Read json file in spark and create dataframe.

val path = "examples/src/main/resources/people.json"
val people = sqlContext.read.json(path)

Save the dataframe using spark-csv

people.write
    .format("com.databricks.spark.csv")
    .option("header", "true")
    .save("newcars.csv")

Source :

read json

save to csv

bob
  • 4,595
  • 2
  • 25
  • 35
  • @CodeHunter for nested json, you will have to explode the json array, take a look at this ans: https://stackoverflow.com/a/45179056/5019163 – bob Jul 10 '18 at 15:23