1

I have a JSON file with contents that look like this:

{
  "dates_included":["2016-01-29"],
  "data":[
    {"field1":"2016-01-29","field2":"3 12 -72"},
    {"field1":"2016-01-29","field2":"33 12 -72"}
  ]
} 

I am trying to create a new DataFrame that contains field1 and field2 as the two columns of the DataFrame.

I can load the JSON file into a DataFrame and register a temp table. But unable to extract field1 and field2 into a new DataFrame.

Intial DataFrame schema looks like this:

root
 |-- data: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- field1: string (nullable = true)
 |    |    |-- field2: string (nullable = true)
 |-- dates_included: array (nullable = true)
 |    |-- element: string (containsNull = true) 
DJElbow
  • 3,345
  • 11
  • 41
  • 52

0 Answers0