7

I have a program that reads a bunch of text and analyzes it. The text may be in any language, but I need to test for japanese and chinese specifically to analyze them a different way.

I have read that I can test each character on it's unicode number to find out if it is in the range of CJK characters. This is helpful, however I would like to separate them if possible to process the text against different dictionaries. Is there a way to test if a character is Japanese OR Chinese?

Suragch
  • 484,302
  • 314
  • 1,365
  • 1,393
landyman
  • 2,023
  • 2
  • 16
  • 15
  • 1
    Do you know the code set, or do you have to guess that too? – Jonathan Leffler Apr 24 '09 at 16:36
  • If you don't know the code set, it may actually make your life easier rather than having everything in unicode. – Elijah Apr 24 '09 at 16:53
  • I wind up converting everything to unicode anyway for analyzing (I'm forced to, really). I can detect the codeset before the conversion; this question is more if the codeset is already unicode. – landyman Apr 24 '09 at 16:59
  • As an addition to this question. What if you need to detect if a character is Chinese or Japanese. It doesn't matter which of the two it is. I am currently trying to match anything in `\p{Han}\p{Hiragana}\p{Katakana}` but the following characters are not matching: 发同讲说宅电的手机告的世全所回广讲说跟 – yarian Jul 13 '11 at 21:58

6 Answers6

6

You won't be able to test a single character to tell with certainty that it is Japanese or Chinese because of the way the unihan code points are implemented in the Unicode standard. Basically, every Chinese character is a potential Japanese character. However, the reverse is not true. Also, there are a number of conventions that could be used to test to see if a block of text is in one language or the other.

  1. Simplifications - if the character you are testing is a PRC simplification such as 门 is only available in main land Chinese.
  2. Kana - if the character is one of the many Japanese kana characters such as あいうえお　then the text block you are working with is definitely Japanese.

The problem arises with the sheer number of characters and words that are in common. However, if I needed a quick and dirty solution to this problem, I would check my entire blocks of text for kana - if the text contains kana then I know it is Japanese. If you need to distinguish Korean as well, I would test for Hangul. Also, if you need to distinguish what type of Chinese, testing for types of simplifications would be the best approach.

Elijah
  • 13,368
  • 10
  • 57
  • 89
  • Thanks. I knew that some of the characters were shared, but I didn't realize that the kana symbols were not part of chinese. I was able to do a test for them that works so far. We'll see how accurate it is after more testing. Thanks again. – landyman Apr 24 '09 at 21:47
  • Simplified Chinese isn't generally found in Japanese Kanji. To differentiate between the two, you could parse a string for the presence of Kana (Hiragana & Katakana), which is specific to Japanese. – Michael Jan 21 '11 at 14:41
2

The process of developing Unicode included the Han Unification. This is because a lot of the Japanese characters are derived from, or the same as, Chinese characters; similarly with Korean. There are some characters (katakana and hiragana - see chapter 12 of the Unicode standard v5.1.0) commonly used in Japanese that would indicate that the text was Japanese rather than Chinese, but I believe it would be a statistical test rather than definitive.

Check out the O'Reilly book on CJKV Information Processing (CJKV is short for Chinese, Japanese, Korean, Vietnamese; I have the CJK predecessor lurking somewhere). There's also the O'Reilly book on Unicode Explained which may be some help, though probably not for this question (I don't recall a discussion of how to identify Japanese and Chinese text).

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
  • Thanks. I'll be sure to check out the books. I agree that testing for only Japanese (hiragana and katakana) isn't definitive, but it'll work for now. – landyman Apr 24 '09 at 21:48
1

You probably can't do that reliably. Japanese uses a lot of the same characters as Chinese. I think the best you could do is to look at a block of text. If you see any uniquely Japanese characters, then you can assume the whole block is Japanese. If not, then it's probably Chinese.

However, I'm just learning Chinese, so I'm not an expert.

Don Kirkby
  • 53,582
  • 27
  • 205
  • 286
0

A workaround is to check the encoding before it is converted to Unicode.

soycamo
  • 78
  • 6
0

There are many characters which are only (commonly) used in Japanese or only used in Chinese.

Japan and China both simplified many characters but often in different ways. You can check for Japanese Shinjitai and Simplified Chinese characters. There are many more of the latter than the former. If there are none of either then you probably have Traditional Chinese.

Of course if you're dealing with Unicode text you may find occasional rare characters or mixed languages which could throw off a heuristic so you're better off going with counting the types of characters to make a judgement.

A good way to find out which characters are common in one language and not in the others is to compare the legacy encodings against each other. You can find mappings of each to Unicode easily on the internet.

I used to have some code I wrote which did a binary search by codepoint and it was extremely fast even in JavaScript - I may have lost it in my travels though (-:

hippietrail
  • 15,848
  • 18
  • 99
  • 158
0

testing for characters in the katakana or hiragana ranges should be a very reliable means of determining whether or not the text is Japanese, especially if you are dealing with 'regular' user-generated text. if you are looking at legal documents or other more official fare it might be slightly more difficult, as there will be a much greater preponderance of complex chinese characters - but it should still be pretty reliable.

si28719e
  • 2,135
  • 5
  • 20
  • 22