Pyspark using lpad not accepting length function

Asked Apr 21 '20 at 21:06

Active Apr 22 '20 at 12:21

Viewed 380 times

I am trying to add a prefix to a string column 'data' value with:

`df.select('data_id',lpad(df['data'],length(df.data) +3 ,'p1/'))`

But I got this error:

TypeError: 'Column' object is not callable

I tried casting with int(length(df.data)), but the message was

TypeError: int() argument must be a string or a number, not 'Column'

I also tried .cast("integer") and I got a similar error.

Additionally, I need to add the prefix just under a condition where 'data_id' is under an interval, like [1 .. 100]. Can you help me?

edited Apr 22 '20 at 12:21

marc_s

asked Apr 21 '20 at 21:06

Stefani

try `df.select('data_id', F.expr("""lpad(data,length(data)+3,'p1/'))"""))` – murtihash Apr 21 '20 at 21:44
I got a "ParseException: u"\nextraneous input ')' ..." – Stefani Apr 21 '20 at 22:13
there was an extra bracket by mistake. try this `df.select('data_id', F.expr("""lpad(data,length(data)+3,'p1/')"""))` – murtihash Apr 21 '20 at 22:18
also import functions like, from `pyspark.sql import functions as F` or `from pyspark.sql.functions import expr` – murtihash Apr 21 '20 at 22:19
for what u want u could also do, `df.select("data_id",F.concat(F.lit('p1/'),F.col("data")))` – murtihash Apr 21 '20 at 22:24
1

@MohammadMurtazaHashmi it works with `concat`. Thanks! – Stefani Apr 21 '20 at 22:35

0 Answers0