How to replace accented characters in PySpark?

Question

I have a string column in a dataframe with values with accents, like

'México', 'Albânia', 'Japão'

How to replace letters with accents to get this:

'Mexico', 'Albania', 'Japao'

I tried many solutions available in Stack OverFlow, like this:

def strip_accents(s):
   return ''.join(c for c in unicodedata.normalize('NFD', s)
                  if unicodedata.category(c) != 'Mn')

But disappointed returns

strip_accents('México')
>>> 'M?xico'

Check this out. https://stackoverflow.com/questions/3322152/is-there-a-way-to-get-rid-of-accents-and-convert-a-whole-string-to-regular-lette — Shavk with a Hoon, Jun 30 '22 at 20:09

Luiz Viola · Accepted Answer · 2022-07-01T07:42:37.223

2

You can use translate:

df = spark.createDataFrame(
    [
    ('1','Japão'),
    ('2','Irã'),
    ('3','São Paulo'),
    ('5','Canadá'),
    ('6','Tókio'),
    ('7','México'),
    ('8','Albânia')
    ],
    ["id", "Local"]
)

df.show(truncate = False)

+---+---------+
|id |Local    |
+---+---------+
|1  |Japão    |
|2  |Irã      |
|3  |São Paulo|
|5  |Canadá   |
|6  |Tókio    |
|7  |México   |
|8  |Albânia  |
+---+---------+

from pyspark.sql import functions as F

df\
    .withColumn('Loc_norm', F.translate('Local',
                                       'ãäöüẞáäčďéěíĺľňóôŕšťúůýžÄÖÜẞÁÄČĎÉĚÍĹĽŇÓÔŔŠŤÚŮÝŽ',
                                       'aaousaacdeeillnoorstuuyzAOUSAACDEEILLNOORSTUUYZ'))\
    .show(truncate=False)

+---+---------+---------+
|id |Local    |Loc_norm |
+---+---------+---------+
|1  |Japão    |Japao    |
|2  |Irã      |Ira      |
|3  |São Paulo|Sao Paulo|
|5  |Canadá   |Canada   |
|6  |Tókio    |Tokio    |
|7  |México   |Mexico   |
|8  |Albânia  |Albânia  |
+---+---------+---------+

edited Jul 01 '22 at 07:42

answered Jul 01 '22 at 07:37

Luiz Viola

2,143
1
11
30

This work! But replace thhe letters e for a, example México to Maxico, lol. But, I think my problem is in the config on aws cluster – Ana Beatriz Jul 06 '22 at 17:25
Maybe the encoding of your file is not UTF-8 – Luiz Viola Jul 07 '22 at 07:12
Why the Albânia continue with accent? – Vivian Mar 02 '23 at 19:53
Seems that ```â``` is not on the list... – Luiz Viola Mar 03 '23 at 08:19
i like this one solution, – JAGJ jdfoxito Apr 19 '23 at 03:30

ZygD · Answer 2 · 2023-06-14T05:53:46.240

In PySpark, you can create a pandas_udf which is vectorized, so it's preferred to a regular udf.

This seems to be the best way to do it in pandas. So, we can use it to create a pandas_udf for PySpark application.

from pyspark.sql import functions as F
import pandas as pd

@F.pandas_udf('string')
def strip_accents(s: pd.Series) -> pd.Series:
    return s.str.normalize('NFKD').str.encode('ascii', 'ignore').str.decode('utf-8')

Test:

df = spark.createDataFrame([('México',), ('Albânia',), ('Japão',)], ['country'])

df = df.withColumn('country2', strip_accents('country'))

df.show()
# +-------+--------+
# |country|country2|
# +-------+--------+
# | México|  Mexico|
# |Albânia| Albania|
# |  Japão|   Japao|
# +-------+--------+

How to replace accented characters in PySpark?

2 Answers2