Add empty column to dataframe in Spark with python

Question

I have a dataframe that i want to make a unionAll with a nother dataframe. The problem is that the second dataframe has thre more columns than the first one. Is there a way for me to add three columns with only empty cells in my first dataframe?

Possible duplicate of [Add an empty column to spark DataFrame](http://stackoverflow.com/questions/33038686/add-an-empty-column-to-spark-dataframe) — zero323, Feb 09 '16 at 15:40

Vishwajeet Pol · Answer 1 · 2020-07-15T17:00:35.403

2

Maybe this will help

To add string type column:

from pyspark.sql.types import StringType

df.withColumn("COL_NAME", lit(None).cast(StringType()))

To Add integer type

from pyspark.sql.types import IntegerType

df.withColumn("COL_NAME", lit(0).cast(IntegerType()))

edited Jul 15 '20 at 17:00

answered Aug 23 '18 at 05:57

Vishwajeet Pol

43
7

1

Just an observation: in your second example, the imported type does not match with the converted type used. – Itagyba Abondanza Kuhlmann Jul 13 '20 at 19:28

score 1 · Answer 2 · answered Feb 09 '16 at 12:51

1

use withColumn object of spark-dataframe
DF.withColumn("NewCol","Value")

answered Feb 09 '16 at 12:51

Hossein Vatani

1,381
14
26

How do I get the value to be a blanc column? – dagrun Feb 09 '16 at 13:10
got it to work with .withColumn('NewCol', lit(None).cast(StringType())) – dagrun Feb 09 '16 at 13:20
I could not understand you. – Hossein Vatani Feb 09 '16 at 13:57

score 1 · Accepted Answer · answered Feb 09 '16 at 13:21

1

df.withColumn('NewColumn', lit(None).cast(StringType()))

answered Feb 09 '16 at 13:21

dagrun

621
3
11
29

If I do df.fillna(0) after this statement. It doesn't work – Akhil Batra Sep 28 '18 at 09:02

Add empty column to dataframe in Spark with python

3 Answers3