withcolumn() in spark for adding new column is not showing result

Question

I am trying to add a new column using withcolumn whose value should be NULL but its not working.

 val schema = StructType(
                    StructField("uid",StringType,true)::
                    StructField("sid",StringType,true)::
                    StructField("astid",StringType,true)::
                    StructField("timestamp",StringType,true)::
                    StructField("start",StringType,true)::
                    StructField("end",StringType,true)::
                    StructField("geo",StringType,true)::
                    StructField("stnid",StringType,true)::
                    StructField("end_type",LongType,true)::
                    StructField("like",LongType,true)::
                    StructField("dislike",LongType,true)::Nil
                      ) 

val Mobpath = spark.read.schema(schema).csv("/data/mob.txt")  
Mobpath.printSchema()

Mobpath.createOrReplaceTempView("Mobpathsql")

val showall =  spark.sql("select * from Mobpathsql")
showall.show()

val newcol = Mobpath.withColumn("new1",functions.lit("null"))
newcol.show()

using withcolumn it is not showing any error and also not showing any output.

It works for me, so I think the other comment may well be valid, but I have some data. — thebluephantom, Nov 04 '18 at 11:33

f.ald · Accepted Answer · 2018-11-04T14:58:17.720

-1

what about this:

val newcol = showall.withColumn("new1",functions.lit("null"))
newcol.show()

I just test the above code and it worked, i don't know why it does not work with Mobpath

edited Nov 04 '18 at 14:58

answered Nov 04 '18 at 12:24

f.ald

320
1
16

1

Not sure I get the question or the answer in fact. – thebluephantom Nov 04 '18 at 14:18
added more detail – f.ald Nov 04 '18 at 14:58

withcolumn() in spark for adding new column is not showing result

1 Answers1