Taking this schema(json dataframe) as an example:
root
|-- abstract: string (nullable = true)
|-- adx_keywords: string (nullable = true)
|-- asset_id: long (nullable = true)
|-- byline: string (nullable = true)
|-- column: string (nullable = true)
|-- des_facet: array (nullable = true)
| |-- element: string (containsNull = true)
|-- eta_id: long (nullable = true)
|-- geo_facet: array (nullable = true)
| |-- element: string (containsNull = true)
|-- id: long (nullable = true)
|-- media: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- approved_for_syndication: long (nullable = true)
| | |-- caption: string (nullable = true)
| | |-- copyright: string (nullable = true)
| | |-- **media-metadata**: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- format: string (nullable = true)
| | | | |-- height: long (nullable = true)
| | | | |-- url: string (nullable = true)
| | | | |-- width: long (nullable = true)
| | |-- subtype: string (nullable = true)
| | |-- type: string (nullable = true)
|-- nytdsection: string (nullable = true)
|-- org_facet: array (nullable = true)
| |-- element: string (containsNull = true)
|-- per_facet: array (nullable = true)
| |-- element: string (containsNull = true)
|-- published_date: string (nullable = true)
|-- section: string (nullable = true)
here I would like to get each array type as separate data frame. I am able to segregate the first levels such as des_facet,geo_facet and media and put it into individual data-frames but unable to do the same for second level for eg: media-metadata in media(array type). A solution to extract nested arrays as an individual dataframe would be helfull
Thanks in advance