1

In my notebook, I have Data Frames being read in that will have a variable number of columns every time the notebook is ran. How do I dynamically change the data types of only the columns that are Boolean data types to String data type?

This is a problem I faced so I am posting the answer incase this helps someone else.

The name of the data frame is "df".

Here we dynamically convert every column in the incoming dataset that is a Boolean data type to a String data type:

def bool_col_DataTypes(DataFrame):

    """This Function accepts a Spark Data Frame as an argument. It returns a list of all Boolean columns in your dataframe."""
    
    DataFrame = dict(DataFrame.dtypes)
    list_of_bool_cols_for_conversion = [x for x, y in DataFrame.items() if y == 'boolean']
    return list_of_bool_cols_for_conversion


list_of_bool_columns = bool_col_DataTypes(df)
    
for i in list_of_bool_columns:
    df = df.withColumn(i, F.col(i).cast(StringType()))
    
new_df = df
JTD2021
  • 127
  • 2
  • 12

1 Answers1

1
data=([(True, 'Lion',1),
       (False, 'fridge',2),
     ( True, 'Bat', 23)])

schema =StructType([StructField('Answer',BooleanType(), True),StructField('Entity',StringType(), True),StructField('ID',IntegerType(), True)])

df=spark.createDataFrame(data, schema)
df.printSchema()

Schema

root
 |-- Answer: boolean (nullable = true)
 |-- Entity: string (nullable = true)
 |-- ID: integer (nullable = true)

Transformation

df1 =df.select( *[col(x).cast('string').alias(x) if y =='boolean' else col(x) for x, y in df.dtypes])

df1.printSchema()

root
 |-- Answer: string (nullable = true)
 |-- Entity: string (nullable = true)
 |-- ID: integer (nullable = true)
wwnde
  • 26,119
  • 6
  • 18
  • 32
  • This is so much better than mine, thank you! I like that you got it all in one line, I was trying something similar to this and it wasn't working but now that I see your answer, I see what I was doing wrong! Also, how did you know to use the "*" in front of the list comprehension? – JTD2021 Sep 10 '22 at 00:55