Filtering nested arrays based on values PySpark

Asked Nov 29 '18 at 13:54

Active Nov 29 '18 at 14:51

Viewed 360 times

I'm trying to filter GA Sessions on PySpark based on the customDimensions. The data is like

+--------------------+--------------------+                                     
|       fullVisitorId|                  cd|
+--------------------+--------------------+
| 5823179578207509663|[[1, app_tv], [36...|
| 5220700153870728639|[[107, live], [10...|
|16421406313456036559|[[1, app_tv], [36...|
|18135892068782985696|[[1, app_tv], [36...|
| 5865612025708664451|[[1, app_tv], [36...|
| 8103574485485735385|[[1, web], [36, d...|
| 6603732532553270294|[[1, web], [36, m...|
|   70498423600813735|[[1, web], [36, d...|
| 5017675391641460547|[[1, web], [36, d...|
+--------------------+--------------------+

Using the GA Schema, the cd (customDimensions) column has an array containing several tuples of index, value pairs.

How can I, efficiently, select the fullVisitorIds that has, for example, an entry with index = 107 and value = 'live' like in the second entry on the example

edited Nov 29 '18 at 14:51

asked Nov 29 '18 at 13:54

waaat

1

Could you check [How to make good reproducible Apache Spark Dataframe examples](https://stackoverflow.com/q/48427185) and [edit] your question accordingly? – zero323 Nov 29 '18 at 14:22
tks, i edited and expanded a little – waaat Nov 29 '18 at 14:51
1

Try exploding cd and do a filter on the exploded column. – pvy4917 Nov 29 '18 at 15:50

Filtering nested arrays based on values PySpark

0 Answers0