Pyspark - exploding dataframe based on values within a list

Question

Here is my dataframe:

+--------------------+--------------------+
|             core_id|    movie_genres_upd|
+--------------------+--------------------+
|12f99f04-5168-438...|[Comedy, Mockumen...|
|32c7d12f-6bf2-4e5...|[Action, Blockbus...|
|9f067041-3b49-4db...|[Animation, Comed...|
|c6d203cb-afcf-4e8...|[Action, Adventur...|
|b02416f9-5761-48f...|[Adventure, Anima...|

These are my data types:

[('core_id', 'string'), ('movie_genres_upd', 'array<string>')]

I'll provide a more visible example. Here is the initial dataframe:

id  genres
1   ["comedy", "blockbuster"]
2   ["drama", "animation", "comedy"] 

Desired dataframe:

id genres
1  "comedy"
1  "blockbuster"
2  "drama"
2  "animation"
2  "comedy"

I'm new to pyspark so i'm struggling with this. Any help would be much appreciated.

Answer 1

Let me know if this helps:

>>> from pyspark.sql.functions import explode
>>> from pyspark.sql.types import (
...     StringType,
...     StructField,
...     StructType,
...     ArrayType
... )
>>>
>>> schema = StructType([
...     StructField('core_id', StringType(), True),
...     StructField('movie_genres_upd', ArrayType(StringType()), True)
... ])
>>>
>>> list = [[1, ["comedy", "blockbuster"]], [2, ["drama", "animation", "comedy"]]]
>>> df = spark.createDataFrame(list, schema)
>>> df2 = df.select('core_id', explode("movie_genres_upd").alias('genre'))
>>> df2.show()
+-------+-----------+
|core_id|      genre|
+-------+-----------+
|      1|     comedy|
|      1|blockbuster|
|      2|      drama|
|      2|  animation|
|      2|     comedy|
+-------+-----------+