'Column' object is not callable with Regex and Pyspark

Question

I need to extract the integers only from url stings in the column "Page URL" and append those extracted integers to a new column. I am using PySpark. My code below:


from pyspark.sql.functions import col, regexp_extract

spark_df_url.withColumn("new_column", regexp_extract(col("Page URL"), "\d+", 1).show())

I have the following error: TypeError: 'Column' object is not callable.

There must be `0`, not `1`, `.withColumn("new_column", regexp_extract("Page URL", "\d+", 0))` — Wiktor Stribiżew, Jan 22 '20 at 18:38
Yes, that's a typo, but even if the syntax was correct, the issue would still be there. — Wiktor Stribiżew, Jan 23 '20 at 08:13

score 2 · Accepted Answer · answered Jan 22 '20 at 18:45

You may use

spark_df_url.withColumn("new_column", regexp_extract("Page URL", "\d+", 0))

Specify the name of the string column as the first argument to regexp_replace and make sure the third argument is set to 0 as your pattern has no capturing groups and you are interested in getting the whole match value as a result.

Note that when you specified 1 as the third argument, you got empty results:

If the regex did not match, or the specified group did not match, an empty string is returned.

'Column' object is not callable with Regex and Pyspark

1 Answers1

Related