RegexOptions.CultureInvariant not finding matches for accents

Question

I would like to create a regex that ignores accent.

For instance:

string s = "I am an old élephant";
string pattern = "elephant";
bool result = new Regex(pattern, RegexOptions.CultureInvariant).IsMatch(s);

My culture when I test is:

System.Globalization.CultureInfo.CurrentCulture = Fr-fr

So I would have expected this code to find a match but it does not.

Is there an easy way to get a match for this?

I am trying to make a StringReplace overload method that would replace élèphânt with elephant and so on.

Answer 1

Use following method:

    public string removeDiacritics(string str)
    {
        var sb = new StringBuilder();

        foreach (char c in str.Normalize(NormalizationForm.FormD))
        {
            if (CharUnicodeInfo.GetUnicodeCategory(c) != UnicodeCategory.NonSpacingMark)
            {
                sb.Append(c);
            }
        }
        return sb.ToString().Normalize(NormalizationForm.FormC);
    }

Then it works

        string s = "I am an old élephant";
        string pattern = "elephant";
        bool result = new Regex(pattern, RegexOptions.IgnoreCase).IsMatch(removeDiacritics(s)); //true

If you have to replace something e.g. iterate (backward) through the matchcollection and edit you original string depending on the indexes of each match.

Explaination: (i'm using the "I am an old élephant" string)

Let's write all chars of the original string into an list :

foreach (char c in str)
{
    chars1.Add(c);
}

As you can see the char is defined as unicode char 233 or 00E9 (see http://unicode-table.com/de/#00E9)

The normalisation is explained here https://msdn.microsoft.com/en-us/library/system.text.normalizationform(v=vs.110).aspx

As the documention says: Form D:

Indicates that a Unicode string is normalized using full canonical decomposition.

That means that the char é is "split up" into an e and an accent char.

To check that, let's output the chars of the normalised string:

List<char> chars2 = new List<char>();
foreach(char c in str.Normalize(NormalizationForm.FormD))
{
    chars2.Add(c);
}

As seen in the watch, the é is now normalised into 2 characters (101 (\u0065) + 769 (\u0301))

Now we have to eliminate these accents: Iterate through all chars of the normalised string and if it's a "NonSpacingMark", add it to the StringBuilder.

MSDN: https://msdn.microsoft.com/en-us/library/system.globalization.unicodecategory(v=vs.110).aspx

NonSpacingMark

Nonspacing character that indicates modifications of a base character. Signified by the Unicode designation "Mn" (mark, nonspacing). The value is 5.

Finally to ensure that all other characters, that are now defined as 2 or 3 characters in our string, are getting "converted" into the unicode character symbol, we have to normalise our new string back to the FormC.

MSDN: FormC:

Indicates that a Unicode string is normalized using full canonical decomposition, followed by the replacement of sequences with their primary composites, if possible.

Answer 2

You are specifying a CultureInvariant regex. That means your culture is ignored. So you either have to remove the option...

bool result = new Regex(pattern).IsMatch(s);

...or if you want to be culture independent, expand your pattern:

string pattern = "[eé]lephant";

Answer 3

If you want to use Regular Expression, you can employ \P{L} to state that a given unicode chracter is a letter.

        string s = "I am an old ùûüÿàâçéèêëïîô";
        string pattern = @"(\p{L})";
        var regex = new Regex(pattern);
        var result = regex.Replace(s, @"$1");
        Console.WriteLine(result);//I am an old uuuyaaceeeeiio