I am trying to scrap a unicode string using javascript. Said string could countain mixed characters. Example: 我的中文不好。我是意大利人。你知道吗？
Ultimately, the string may contain - Chinese characters - Chinese punctuation - ANSI characters and punctuation
I need to leave the Chinese characters only . Any hint ?
You can see the relevant blocks at http://www.unicode.org/reports/tr38/#BlockListing or http://www.unicode.org/charts/ .
If you are excluding compatibility characters (ones which should no longer be used), as well as strokes, radicals, and Enclosed CJK Letters and Months, the following ought to cover it (I've added the individual JavaScript equivalent expressions afterward):
[\u4E00-\u9FCC][\u3400-\u4DB5][\ud840-\ud868][\udc00-\udfff]|\ud869[\udc00-\uded6]\ud869[\udf00-\udfff]|[\ud86a-\ud86c][\udc00-\udfff]|\ud86d[\udc00-\udf34]\ud86d[\udf40-\udfff]|\ud86e[\udc00-\udc1d][\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29]...so, a regex to grab the Chinese characters would be:
/[\u4E00-\u9FCC\u3400-\u4DB5\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29]|[\ud840-\ud868][\udc00-\udfff]|\ud869[\udc00-\uded6\udf00-\udfff]|[\ud86a-\ud86c][\udc00-\udfff]|\ud86d[\udc00-\udf34\udf40-\udfff]|\ud86e[\udc00-\udc1d]/
Due in fact to the many CJK (Chinese-Japanese-Korean) characters, Unicode was expanded to handle more characters beyond the "Basic Multilingual Plane" (called "astral" characters), and since the CJK Unified Ideographs extensions B-D are examples of such astral characters, those extensions have ranges that are more complicated because they have to be encoded using surrogate pairs in UTF-16 systems like JavaScript. A surrogate pair consists of a high surrogate and a low surrogate, neither of which is valid by itself but when joined together form an actual single character despite their string length being 2).
While it would probably be easier for replacement purposes to express this as the non-Chinese characters (to replace them with the empty string), I provided the expression for the Chinese characters instead so that it would be easier to track in case you needed to add or remove from the blocks.
Update September 2017
As of ES6, one may express the regular expressions without resorting to surrogates by using the "u" flag along with the code point inside of the new escape sequence with brackets, e.g., /^[\u{20000}-\u{2A6D6}]*$/u for "CJK Unified Ideographs Extension B".
Note that Unicode too has progressed to include "CJK Unified Ideographs Extension E" ([\u{2B820}-\u{2CEAF}]) and "CJK Unified Ideographs Extension F" ([\u{2CEB0}-\u{2EBEF}]).
For ES2018, it appears that Unicode property escapes will be able to simplify things even further. Per http://2ality.com/2017/07/regexp-unicode-property-escapes.html , it looks like will be able to do:
/^(\p{Block=CJK Unified Ideographs}|\p{Block=CJK Unified Ideographs Extension A}|\p{Block=CJK Unified Ideographs Extension B}|\p{Block=CJK Unified Ideographs Extension C}|\p{Block=CJK Unified Ideographs Extension D}|\p{Block=CJK Unified Ideographs Extension E}|\p{Block=CJK Unified Ideographs Extension F}|[\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29])+$/u
And as the shorter aliases from http://unicode.org/Public/UNIDATA/PropertyAliases.txt and http://unicode.org/Public/UNIDATA/PropertyValueAliases.txt can also be used for these blocks, you could shorten this to the following (and changing underscores to spaces or casing apparently too if desired):
/^(\p{blk=CJK}|\p{blk=CJK_Ext_A}|\p{blk=CJK_Ext_B}|\p{blk=CJK_Ext_C}|\p{blk=CJK_Ext_D}|\p{blk=CJK_Ext_E}|\p{blk=CJK_Ext_F}|[\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29])+$/u
And if we wanted to improve readability, we could document the falsely labeled compatibility characters using named capture groups (see http://2ality.com/2017/05/regexp-named-capture-groups.html ):
/^(\p{blk=CJK}|\p{blk=CJK_Ext_A}|\p{blk=CJK_Ext_B}|\p{blk=CJK_Ext_C}|\p{blk=CJK_Ext_D}|\p{blk=CJK_Ext_E}|\p{blk=CJK_Ext_F}|(?<CJKFalseCompatibilityUnifieds>[\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29]))+$/u
And as it looks per http://unicode.org/reports/tr44/#Unified_Ideograph like the "Unified_Ideograph" property (alias "UIdeo") covers all of our unified ideographs and excluding symbols/punctuation and compatibility characters, if you don't need to pick and choose out of the above, the following may be all you need:
/^\p{Unified_Ideograph=yes}*$/u
or in shorthand:
/^\p{UIdeo=y}*$/u
Update April 2023
It seems that JavaScript has yet to support Block/blk as one of the acceptable property aliases: see Table 65 of https://tc39.es/ecma262/multipage/text-processing.html#table-nonbinary-unicode-properties for the currently allowable non-binary properties.
As of Chrome 64, Firefox 78, Safari 11.1, and Edge 79, the simplest regex to test whether a string is a Chinese character is /\p{Script=Han}/u. The \p{} specifies a Unicode property escape, the Script=Han expression matches any character whose script property is Han (Chinese), and the u flag enables usage of Unicode features in the regex, such as these property escapes.
So you could filter all of the non-Chinese characters from a string like this:
console.log(
"hello! 42 我的中文不好。我是意大利人。你知道吗？"
.split("")
.filter(char => /\p{Script=Han}/u.test(char))
.join("")
);The Script property name can also be abbreviated, as in /\p{sc=Han}/u.
A copy and paste solution. Uses ES6's unicode flag. All current extensions, up to Extension F, and the Ideographs.
const character_xp = new RegExp(String.raw`
[\u{FA0E}\u{FA0F}\u{FA11}\u{FA13}\u{FA14}\u{FA1F}\u{FA21}\u{FA23}\u{FA24}\u{FA27}-\u{FA29}]
|[\u{4E00}-\u{9FCC}]
|[\u{3400}-\u{4DB5}]
|[\u{20000}-\u{2A6D6}]
|[\u{2A700}-\u{2B734}]
|[\u{2B740}-\u{2B81D}]
|[\u{2B820}-\u{2CEAF}]
|[\u{2CEB0}-\u{2EBEF}]
`.replace(/\s+/g, ''), "u")
Rather than inventing your own solution you could probably use unicode-data module (one of the modules generated by it, to be precise), which is essentially a javascript interface to UnicodeData.txt database (akin to unicodedata standard module in python, if it rings your bell).
