0

I am using the microsoft.spark version 1.0.0 to process a parquet file that is received from the network. I have mapped the parquet file into a Dataframe and i am trying to strip the alias from the userid column which is stored as an email address (user testuser is saved as testuser@gmail.com). Can't seem to figure out how to turn Column into an enumerable and there is not built in way to iterate over the column. Any pointers would be helpful

dataframe.WithColumn("alias", ExtractAlias(dataframe["userid"]))
ExtractAlias(Column userID_column){need to iterate over the column to extract out the user alias}
mck
  • 40,932
  • 13
  • 35
  • 50
Bonaii
  • 75
  • 6

1 Answers1

1
using Microsoft.Spark.Sql;

dataframe.WithColumn("alias", Functions.split(Functions.col("userid"), "@").GetItem(0))
mck
  • 40,932
  • 13
  • 35
  • 50