How can I detect certain Unicode characters in a string in Ruby?

Question

Given a string in Ruby 1.8.7 (without the awesome Oniguruma regular expression engine that supports Unicode properties with \p{}), I would like to be able to determine if the string contains one or more Chinese, Japanese, or Korean characters; i.e.

class String
  def contains_cjk?
    ...
  end
end

>> '日本語'.contains_cjk?
=> true
>> '광고 프로그램'.contains_cjk?
=> true
>> '艾弗森将退出篮坛'.contains_cjk?
=> true
>> 'Watashi ha bakana gaijin desu.'.contains_cjk?
=> false

I suspect that this will boil down to seeing if any of the characters in the string are in the Unihan CJKV Unicode blocks, but I figured it was worth asking if anyone knows of an existing solution in Ruby.

Are you using version 1.9 of Ruby, or just an older version without good Unicode regex support? If you’re using 1.9, you should have access to (some) Unicode properties, like `\p{InCJKUnifiedIdeographs}` or maybe even `\p{Han}`. — tchrist, Jan 13 '11 at 14:57

score 49 · Accepted Answer · edited Jan 17 '20 at 14:57

49

(ruby 1.9.2)

#encoding: UTF-8
class String
  def contains_cjk?
    !!(self =~ /\p{Han}|\p{Katakana}|\p{Hiragana}|\p{Hangul}/)
  end
end

strings= ['日本', '광고 프로그램', '艾弗森将退出篮坛', 'Watashi ha bakana gaijin desu.']
strings.each{|s| puts s.contains_cjk?}

#true
#true
#true
#false

\p{} matches a character’s Unicode script.
The following scripts are supported: Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese, Buhid, Canadian_Aboriginal, Carian, Cham, Cherokee, Common, Coptic, Cuneiform, Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic, Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hiragana, Inherited, Kannada, Katakana, Kayah_Li, Kharoshthi, Khmer, Lao, Latin, Lepcha, Limbu, Linear_B, Lycian, Lydian, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko, Ogham, Ol_Chiki, Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician, Rejang, Runic, Saurashtra, Shavian, Sinhala, Sundanese, Syloti_Nagri, Syriac, Tagalog, Tagbanwa, Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Vai, and Yi.

Wow. Ruby Regexp source .

edited Jan 17 '20 at 14:57

Hector Correa

26,290
8
57
73

answered Jan 13 '11 at 15:13

steenslag

79,051
16
138
171

This definitely works in Ruby 1.9, or Ruby 1.8 with the Oniguruma regex engine. I'm using 1.8.7 without Oniguruma, tragically. :( Great solution, even though it does not help me in this particular case. – Josh Glover Jan 13 '11 at 15:21
2

I had to add '# encoding: UTF-8' to the top of the file to make this work. Otherwise I got a invalid character property name error. – Morrowless Sep 06 '11 at 16:01
1

Makes more sense to do `[p{Han}\p{Katakana}\p{Hiragana}\p{Hangul}]`. – tchrist Jun 04 '12 at 02:21
@tchrist that results in `false, true, false, true`. – steenslag Jun 23 '12 at 22:42
1

@steenslag Then there’s a bug, because `/a|b|c|d/` cannot get a different answer from what `[abcd]` gets when each one of those a,b,c,d is a single character or something that matches a single character, like `\w` or `\p{Han}`. Best check it out very very carefully. – tchrist Jun 24 '12 at 01:48
The new source link (Ruby 2.0): https://github.com/ruby/ruby/blob/trunk/doc/regexp.rdoc – Juanito Fatas Jul 20 '13 at 13:08
are there groups, as in p{south_asian}, p{middle_eastern} etc ? or do i have to use multiple OR's to create them ? – amitava mozumder Oct 30 '18 at 11:22

Josh Glover · Answer 2 · 2011-01-13T16:31:09.217

Given my Ruby 1.8.7 constraint, this is the best I could do:

class String
  CJKV_RANGES = [
      (0xe2ba80..0xe2bbbf),
      (0xe2bfb0..0xe2bfbf),
      (0xe38080..0xe380bf),
      (0xe38180..0xe383bf),
      (0xe38480..0xe386bf),
      (0xe38780..0xe387bf),
      (0xe38880..0xe38bbf),
      (0xe38c80..0xe38fbf),
      (0xe39080..0xe4b6bf),
      (0xe4b780..0xe4b7bf),
      (0xe4b880..0xe9bfbf),
      (0xea8080..0xea98bf),
      (0xeaa080..0xeaaebf),
      (0xeaaf80..0xefbfbf),
  ]

  def contains_cjkv?
    each_char do |ch|
      return true if CJKV_RANGES.any? {|range| range.member? ch.unpack('H*').first.hex }
    end
    false
  end
end


strings = ['日本', '광고 프로그램', '艾弗森将退出篮坛', 'Watashi ha bakana gaijin desu.']
strings.each {|s| puts s.contains_cjkv? }

#true
#true
#true
#false

Pretty hacktacular, but it works. It actually detects a variety of Indic scripts as well, so it should probably really be called contains_asian?

Maybe I should gem this up for other poor I18N hackers stuck with Ruby 1.8.

I have a project stuck on 1.8, too. This solution didn't work for me, but I adapted a solution from another Stack Overflow thread – see my answer here. — Henrik N, Dec 14 '13 at 18:03

score 1 · Answer 3 · answered Jun 04 '12 at 01:26

1

I've written a little gem that packages up the approach in steenslag's answer above:

https://github.com/jpatokal/script_detector

It can also take a stab at differentiating between Japanese, Korean, simplified Chinese and traditional Chinese, although due to the complexities of Han unification it only works reliably with large slabs of text.

answered Jun 04 '12 at 01:26

lambshaanxy

22,552
10
68
92

I wonder if it's the most up-to-date library. – Nakilon Aug 21 '23 at 04:33

score 0 · Answer 4 · edited May 23 '17 at 12:09

0

Ruby 1.8 solution based on this code and using the API from Josh Glover's solution on this thread:

class String
  CJKV_RANGES = [
    (0x4E00..0x9FFF),
    (0x3400..0x4DBF),
    (0x20000..0x2A6DF),
    (0x2A700..0x2B73F),
  ]

  def contains_cjkv?
    unpack("U*").any? { |char|
      CJKV_RANGES.any? { |range| range.member?(char) }
    }
  end
end

edited May 23 '17 at 12:09

Community

1
1

answered Dec 14 '13 at 18:01

Henrik N

15,786
5
82
131

How can I detect certain Unicode characters in a string in Ruby?

4 Answers4

Linked