Generate empty spark DF provided a list with column names

Question

I am not sure if this is a valid question but I would like to ask.

Is there a way that i can use a list with column names and generate an empty spark dataframe, the schema should be created with the elements from the list with the datatype for all columns as StringType.

e.g:

column_names = "ColA|ColB|ColC"

def Convert(string):
    li = list(string.split("|"))
    return li

schema_names = Convert(column_names)

#schema_names = ['ColA', 'ColB', 'ColC']

How can I use this list to create a DF Schema or an empty DF

**This is somewhat similar to How to create an empty DataFrame with a specified schema? , as I am also trying to create empty DF schema, but the approach I am mentioned is different. I am trying to generate the schema from the list.

Possible duplicate of [How to create an empty DataFrame with a specified schema?](https://stackoverflow.com/questions/31477598/how-to-create-an-empty-dataframe-with-a-specified-schema) — Chandan Ray, Sep 18 '18 at 10:24
@ChandanRay the way you do that in python and in scala is totally different. — Steven, Sep 18 '18 at 12:12

score 9 · Answer 1 · answered Sep 18 '18 at 16:18

Since you want all columns to be StringType(), define the schema as follows:

from pyspark.sql.types import StructType, StructField, StringType

column_names = "ColA|ColB|ColC"
mySchema = StructType([StructField(c, StringType()) for c in column_names.split("|")])

Now just pass in an empty list as the data along with this schema to spark.createDataFrame():

df = spark.createDataFrame(data=[], schema=mySchema)
df.show()
#+----+----+----+
#|ColA|ColB|ColC|
#+----+----+----+
#+----+----+----+

Now you can reuse this schema for other DataFrames as well.

score 4 · Accepted Answer · answered Sep 18 '18 at 12:15

4

I have a dirty solution. Probably not the best one :

column_names = "ColA|ColB|ColC"

df = spark.createDataFrame(
  [
    tuple('' for i in column_names.split("|"))
  ],
  column_names.split("|")
).where("1=0")

df.show()

+----+----+----+
|ColA|ColB|ColC|
+----+----+----+
+----+----+----+

answered Sep 18 '18 at 12:15

Steven

14,048
6
38
73

is there a way that i can use this schema for other dataframes? I was actually trying to dynamically create the dataframe schema so that i can map to my DF – darkmatter Sep 18 '18 at 12:55
1

sure. `spark.createDataframe( [some_data], df.schema)` – Steven Sep 18 '18 at 13:01
it erros out..`TypeError: schema should be StructType or list or None, but got: DataFrame` – darkmatter Sep 18 '18 at 13:03
you probably did something wrong cause df.schema is no dataframe. it is structype. – Steven Sep 18 '18 at 13:04
yeah, i was passing the df, instead of df.schema – darkmatter Sep 18 '18 at 13:06
@pault Wrong. `df.count() --> 0` – Steven Sep 18 '18 at 16:07
Sorry, missed the `where` statement. There's a simpler way. – pault Sep 18 '18 at 16:09
@pault Im sure there is probably a cleaner/simpler way. Please show us. I would like to see it – Steven Sep 18 '18 at 16:10
@Steven you can pass in an empty list as the data and specify the data types using a `StructType` for the schema. – pault Sep 18 '18 at 16:22

score 2 · Answer 3 · answered Sep 18 '18 at 12:31

In Scala:

val columns = List("ColA", "ColB", "ColC")
val result = columns.foldLeft(spark.emptyDataFrame)((a, b) => a.withColumn(b, lit("anyStringValue")))
result.printSchema()
result.show(false)

Output:

root
 |-- ColA: string (nullable = false)
 |-- ColB: string (nullable = false)
 |-- ColC: string (nullable = false)

+----+----+----+
|ColA|ColB|ColC|
+----+----+----+
+----+----+----+

Generate empty spark DF provided a list with column names

3 Answers3