I have a JSON file with contents that look like this:
{
"dates_included":["2016-01-29"],
"data":[
{"field1":"2016-01-29","field2":"3 12 -72"},
{"field1":"2016-01-29","field2":"33 12 -72"}
]
}
I am trying to create a new DataFrame that contains field1
and field2
as the two columns of the DataFrame.
I can load the JSON file into a DataFrame and register a temp table. But unable to extract field1
and field2
into a new DataFrame.
Intial DataFrame schema looks like this:
root
|-- data: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- field1: string (nullable = true)
| | |-- field2: string (nullable = true)
|-- dates_included: array (nullable = true)
| |-- element: string (containsNull = true)