10

I am trying to match a character and all its possible diacritic variations (aka accent-insensitive) with a regular expression. What I could do of course is:

re.match(r"^[eēéěèȅêęëėẹẽĕȇȩę̋ḕḗḙḛḝė̄]$", "é")

but that is not a general solution. If I use unicode categories like \pL I can't reduce the match to a specific character, in this case e.

Felk
  • 7,720
  • 2
  • 35
  • 65
  • Does Python regex support the *POSIX character equivalence* match? This is as simple as `[=e=]` – but it also matches its capitalized equivalences, which may be too much in your case. This could be worked around using `(?!\u)`, although this in turn needs your regex to support Uppercase for all Unicode characters as well. – Jongware Mar 03 '16 at 23:25

1 Answers1

20

A workaround to achieve the desired goal would be to use unidecode to get rid of all diacritics first, and then just match agains the regular e

re.match(r"^e$", unidecode("é"))

Or in this simplified case

unidecode("é") == "e"

Another solution which doesn't depend on the unidecode-library, preserves unicode and gives more control is manually removing the diacritics as follows:

Use unicodedata.normalize() to turn your input string into normal form D (for decomposed), making sure composite characters like é get turned into the decomposite form e\u301 (e + COMBINING ACUTE ACCENT)

>>> input = "Héllô"
>>> input
'Héllô'
>>> normalized = unicodedata.normalize("NFKD", input)
>>> normalized
'He\u0301llo\u0302'

Then, remove all codepoints which fall into the category Mark, Nonspacing (short Mn). Those are all characters that have no width themselves and just decorate the previous character. Use unicodedata.category() to determine the category. 

>>> stripped = "".join(c for c in normalized if unicodedata.category(c) != "Mn")
>>> stripped
'Hello'

The result can be used as a source for regex-matching, just as in the unidecode-example above. Here's the whole thing as a function:

def remove_diacritics(text):
    """
    Returns a string with all diacritics (aka non-spacing marks) removed.
    For example "Héllô" will become "Hello".
    Useful for comparing strings in an accent-insensitive fashion.
    """
    normalized = unicodedata.normalize("NFKD", text)
    return "".join(c for c in normalized if unicodedata.category(c) != "Mn")
Felk
  • 7,720
  • 2
  • 35
  • 65