Can I change the nullability of a column in my Spark dataframe?

Question

I have a StructField in a dataframe that is not nullable. Simple example:

import pyspark.sql.functions as F
from pyspark.sql.types import *
l = [('Alice', 1)]
df = sqlContext.createDataFrame(l, ['name', 'age'])
df = df.withColumn('foo', F.when(df['name'].isNull(),False).otherwise(True))
df.schema.fields

which returns:

[StructField(name,StringType,true), StructField(age,LongType,true), StructField(foo,BooleanType,false)]

Notice that the field foo is not nullable. Problem is that (for reasons I won't go into) I want it to be nullable. I found this post Change nullable property of column in spark dataframe which suggested a way of doing it so I adapted the code therein to this:

import pyspark.sql.functions as F
from pyspark.sql.types import *
l = [('Alice', 1)]
df = sqlContext.createDataFrame(l, ['name', 'age'])
df = df.withColumn('foo', F.when(df['name'].isNull(),False).otherwise(True))
df.schema.fields
newSchema = [StructField('name',StringType(),True), StructField('age',LongType(),True),StructField('foo',BooleanType(),False)]
df2 = sqlContext.createDataFrame(df.rdd, newSchema)

which failed with:

TypeError: StructField(name,StringType,true) is not JSON serializable

I also see this in the stack trace:

raise ValueError("Circular reference detected")

So I'm a bit stuck. Can anyone modify this example in a way that enables me to define a dataframe where column foo is nullable?

icarus · Answer 1 · 2018-09-03T08:42:27.930

12

I know this question is already answered, but I was looking for a more generic solution when I came up with this:

def set_df_columns_nullable(spark, df, column_list, nullable=True):
    for struct_field in df.schema:
        if struct_field.name in column_list:
            struct_field.nullable = nullable
    df_mod = spark.createDataFrame(df.rdd, df.schema)
    return df_mod

You can then call it like this:

set_df_columns_nullable(spark,df,['name','age'])

edited Sep 03 '18 at 08:42

answered Aug 13 '18 at 11:35

icarus

281
2
11

1

Great answer. Are there any performance implications in doing this? What exactly happens when you "create a new dataframe" based on the existing RDD? – malthe Mar 02 '21 at 20:28
Is there any way by which this can be done in scala? – Nikunj Kakadiya May 11 '22 at 13:50
4

@malthe and others come here, seems this is super costly, i tested it with a 10X1 df only, and the one without updating the nullable took `184.47592 ms` while with this update nullable in place took `1187.75746 ms` – lnshi May 13 '22 at 05:54

score 9 · Answer 2 · answered Nov 25 '20 at 10:50

9

For the general case, one can change the nullability of a column via the nullable property of the StructField of that specific column. Here's an example:

df.schema['col_1']
# StructField(col_1,DoubleType,false)

df.schema['col_1'].nullable = True

df.schema['col_1']
# StructField(col_1,DoubleType,true)

answered Nov 25 '20 at 10:50

yatu

86,083
12
84
139

2

this seems to only work if i re-create the dataframe with the schema change. is there no way to do this "in-line"? – Grizzly2501 Dec 09 '22 at 09:15
[This SO answer](https://stackoverflow.com/a/46119565/3692004) shows how to change the nullable property in-place. – Benji Feb 02 '23 at 10:19

score 5 · Accepted Answer · answered Sep 06 '17 at 10:53

Seems you missed the StructType(newSchema).

l = [('Alice', 1)]
df = sqlContext.createDataFrame(l, ['name', 'age'])
df = df.withColumn('foo', F.when(df['name'].isNull(),False).otherwise(True))
df.schema.fields
newSchema = [StructField('name',StringType(),True), StructField('age',LongType(),True),StructField('foo',BooleanType(),False)]
df2 = sqlContext.createDataFrame(df.rdd, StructType(newSchema))
df2.show()

score -3 · Answer 4 · answered Sep 06 '17 at 10:21

-3

df1 = df.rdd.toDF()
df1.printSchema()

Output:

root
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)
 |-- foo: boolean (nullable = true)

answered Sep 06 '17 at 10:21

Abhishek Bansal

1,481
1
8
8

Can I change the nullability of a column in my Spark dataframe?

4 Answers4

Linked