How to convert UTF8 combined Characters into single UTF8 characters in ruby?

Question

Some characters such as the Unicode Character 'LATIN SMALL LETTER C WITH CARON' can be encoded as 0xC4 0x8D, but can also be represented with the two code points for 'LATIN SMALL LETTER C' and 'COMBINING CARON', which is 0x63 0xcc 0x8c.
More info here: http://www.fileformat.info/info/unicode/char/10d/index.htm

I wonder if there is a library which can convert a 'LATIN SMALL LETTER C' + 'COMBINING CARON' into 'LATIN SMALL LETTER C WITH CARON'. Or is there a table containing these conversions?

Please do not be thinking about 8-bit code units. Think only of logical code points. You can use `unicode_utils` for converting into NFC form, but that does not do what you want, because what you want is not possible. — tchrist, Aug 04 '11 at 14:42

score 8 · Answer 1 · answered Aug 04 '11 at 05:15

These conversions don't always exist. The combination of U+0063 (c) with U+030C (combining caron) can be represented as a single character, for instance, but there's no precomposed character representing a lowercase 'w' with a caron (w̌).

Nevertheless, there exist libraries which can perform this composition where possible. Look for a Unicode function called "NFC" (Normalization Form: Composition). See, for instance: http://unicode-utils.rubyforge.org/classes/UnicodeUtils.html#M000015

score 7 · Accepted Answer · edited May 17 '19 at 12:12

Generally, you use Unicode Normalization to do this.

Using UnicodeUtils.nfkc using the gem unicode_utils (https://github.com/lang/unicode_utils) should get you the specific behavior you're asking for; unicode normalization form kC will use a compatibility decomposition followed by converting the string to a composed form, if available (basically what you asked for by your example). (You may also get close to what you want with normalization form c, sometimes acronymized NFC).

How to replace the Unicode gem on Ruby 1.9? has additional details.

In Ruby 1.8.7, you'd need do gem install Unicode, for which there is a similar function available.

Edited to add: The main reason why you'll probably want normalization form kC instead of just normalization form C is that ligatures (characters that are squeezed together for historical/typographical reasons) will first be decomposed to the individual characters, which is sometimes desirable if you're doing lexicographic ordering or searching).

You assume that such a precombined character exists. That is not a good assumption. Notice how U+0035 ‹5› `DIGIT FIVE` followed by U+0304 ‹◌̄› `COMBINING MACRON` has no shorter rendering in NFC than it has in NFD. Only a few legacy code points have precombined characters. Most do not. — tchrist, Aug 05 '11 at 03:46

score 0 · Answer 3 · answered Mar 02 '14 at 09:59

String#encode can be used since Ruby 1.9. UTF-8-MAC is a variant of NFD. The codepoints in the range between U+2000 and U+2FFF, or U+F900 and U+FAFF, or U+2F800 and U+2FAFF are not decomposed. See https://developer.apple.com/library/mac/qa/qa1173/_index.html for the details. UTF-8-HFS can be also used insted of UTF-8-MAC.

# coding: utf-8

s = "\u010D"
s.encode!('UTF-8-MAC', 'UTF-8')
s.force_encoding('UTF-8')

p "\x63\xcc\x8c" == s
p "\u0063" == s[0]
p "\u030C" == s[1]

How to convert UTF8 combined Characters into single UTF8 characters in ruby?

3 Answers3

Linked