2

Taking this schema(json dataframe) as an example:

 root
     |-- abstract: string (nullable = true)
     |-- adx_keywords: string (nullable = true)
     |-- asset_id: long (nullable = true)
     |-- byline: string (nullable = true)
     |-- column: string (nullable = true)
     |-- des_facet: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- eta_id: long (nullable = true)
     |-- geo_facet: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- id: long (nullable = true)
     |-- media: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- approved_for_syndication: long (nullable = true)
     |    |    |-- caption: string (nullable = true)
     |    |    |-- copyright: string (nullable = true)
     |    |    |-- **media-metadata**: array (nullable = true)
     |    |    |    |-- element: struct (containsNull = true)
     |    |    |    |    |-- format: string (nullable = true)
     |    |    |    |    |-- height: long (nullable = true)
     |    |    |    |    |-- url: string (nullable = true)
     |    |    |    |    |-- width: long (nullable = true)
     |    |    |-- subtype: string (nullable = true)
     |    |    |-- type: string (nullable = true)
     |-- nytdsection: string (nullable = true)
     |-- org_facet: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- per_facet: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- published_date: string (nullable = true)
     |-- section: string (nullable = true)

here I would like to get each array type as separate data frame. I am able to segregate the first levels such as des_facet,geo_facet and media and put it into individual data-frames but unable to do the same for second level for eg: media-metadata in media(array type). A solution to extract nested arrays as an individual dataframe would be helfull

Thanks in advance

  • Could you update your question providing your Json as data sample? Otherwise we can't test it and answer you precisely – Kafels Sep 19 '21 at 13:29

0 Answers0