Regex pattern to remove numeric value from words in pyspark

Question

I am working on pyspark dataframe and I have a column of words (array<string> type). What should be the regex pattern to remove numeric values and numeric values from words?

+---+----------------------------------------------+
|id |    words                                     |
+---+----------------------------------------------+
|564|[fhbgtrj5, 345gjhg, ghth578ghu, 5897, fhrfu44]|
+---+----------------------------------------------+

expected output:

+---+----------------------------------------------+
|id |words                                         |
+---+----------------------------------------------+
|564|               [fhbgtrj, gjhg, ghthghu, fhrfu]|
+---+----------------------------------------------+

Please help.

Does this answer your question? [Delete digits in Python (Regex)](https://stackoverflow.com/questions/817122/delete-digits-in-python-regex) — jbflow, Mar 25 '21 at 22:27
@jbflow thanks for looking into it. the references you shared certainly removes numbers but another aim is to keep alphabets from alphanumeric — Samiksha, Mar 25 '21 at 22:32

mck · Accepted Answer · 2021-03-26T09:16:27.520

You can use transform together with regexp_replace to remove the numbers, and use array_remove to remove the empty entries (which comes from those entries which only consist of numbers).

df2 = df.withColumn(
    'words', 
    F.expr("array_remove(transform(words, x -> regexp_replace(x, '[0-9]', '')), '') as words")
)

df2.show(truncate=False)
+---+-------------------------------+
|id |words                          |
+---+-------------------------------+
|564|[fhbgtrj, gjhg, ghthghu, fhrfu]|
+---+-------------------------------+

Regex pattern to remove numeric value from words in pyspark

1 Answers1