11

English, of course, is a no-brainer for regex because that's what it was originally developed in/for:

Can regular expressions understand this character set?

French gets into some accented characters which I'm unsure how to match against - i.e. are è and e both considered word characters by regex?

Les expressions régulières peuvent comprendre ce jeu de caractères?

Japanese doesn't contain what I know as regex word characters to match against.

正規表現は、この文字を理解でき、設定?

Alan Moore
  • 73,866
  • 12
  • 100
  • 156
John K
  • 28,441
  • 31
  • 139
  • 229
  • I think this may also depend heavily on the platform on which the regex engine is running, did you have one in mind? – Lazarus Mar 03 '10 at 13:56
  • "Regex", or "regular expression", is a concept defined for any collection of symbols you might want to call an alphabet. In practice, there are many regular expression engines (all of which I've seen add other capabilities as well), some of which presumably handle Unicode of some flavor well and some of which probably don't. In short, this is a platform-dependent question, and to get a useful response you will need to tell us which regex engine you're talking about. – David Thornley Mar 03 '10 at 15:20

8 Answers8

9

Short answer: yes.

More specifically it depends on your regex engine supporting unicode matches (as described here).

Such matches can complicate your regular expressions enormously, so I can recommend reading this unicode regex tutorial (also note that unicode implementations themselves can be quite a mess so you might also benefit from reading Joel Spolsky's article about the inner workings of character sets).

Joachim Sauer
  • 302,674
  • 57
  • 556
  • 614
Lars Tackmann
  • 20,275
  • 13
  • 66
  • 83
3

"[\p{L}]" This regular expression contains all characters that are letters, from all languages, upper and lower case. so letters like (a-z A-Z ä ß è 正 の文字を理解) are accepted but signs like (, . ? > :) or other similar ones are not.

  • the brackets [] mean that this expression is a set.
  • If you want unlimited number of letters from this set to be accepted, use an astrix * after the brackets, like this: "[\p{L}]*"
  • it is always important to make sure you take care of white space in your regex. since your evaluation might fail because of white space. To solve this you can use: "[\p{L} ]*" (notice the white space inside brackets)
  • If you want to include the numbers as well, "[\p{L|N} ]*" can help. p{N} matches any kind of numeric character in any script.
div-ane
  • 41
  • 7
  • I also find this very help full for different languages: https://medium.com/@h2s1880/how-to-use-regular-expressions-to-distinguish-national-languages-in-swift-c19d6d8d0a97 – div-ane Oct 10 '20 at 13:08
1

As far as I know, there isn't any specific pattern you can use i.e. [a-zA-Z] to match "è", but you can always match them in separately, i.e. [a-zA-Zè正]

Obviously that can make your regexp immense, but you can always control this by adding your strings into variables, and only passing the variables into the expressions.

Marcos Placona
  • 21,468
  • 11
  • 68
  • 93
1

Generally speaking, regex is more for grokking machine-readable text than for human-readable text. It is in many ways a more general answer to the whole XML with regex thing; regex is by its very nature incapable of properly parsing human language, because the language is more complex than what you are using to parse it.

If you want to break down human language (English included), you would want to use a language analysis tool or even an AI, not mere regular expressions.

Williham Totland
  • 28,471
  • 6
  • 52
  • 68
1

/[\p{Latin}]/ should for example, include Latin alphabet. You can get the full explanation and reference here.

casraf
  • 21,085
  • 9
  • 56
  • 91
0

it is not about the regular expression but about framework that executes it. java and .net i think are very good in handling unicode. so "è and e both considered word characters by regex" is true.

Andrey
  • 59,039
  • 12
  • 119
  • 163
0

It depends on the implementation and the character set. In general the answer is "Yes," but it may require additional setup on your part.

In Perl, for example, the meaning of things like \w is altered by the chosen locale (use locale).

sorpigal
  • 25,504
  • 8
  • 57
  • 75
0

This SO thread might help. It includes the Unicode character classes you can use in a regex (e.g., [Ll] is all lowercase letters, regardless of language).

Community
  • 1
  • 1
Tom
  • 22,301
  • 5
  • 63
  • 96