Convert Unicode surrogate pair to literal string

Question

I am trying to read a high Unicode character from one string into another. For brevity, I will simplify my code as shown below:

public static void UnicodeTest()
{
    var highUnicodeChar = ""; //Not the standard A

    var result1 = highUnicodeChar; //this works
    var result2 = highUnicodeChar[0].ToString(); // returns \ud835
}

When I assign highUnicodeChar to result1 directly, it retains its literal value of . When I try to access it by index, it returns \ud835. As I understand it, this is a surrogate pair of UTF-16 characters used to represent a UTF-32 character. I am pretty sure this problem has to do with trying to implicitly convert a char to a string.

In the end, I want result2 to yield the same value as result1. How can I do this?

score 32 · Accepted Answer · edited Jun 07 '19 at 05:43

In Unicode, you have code points. These are 21 bits long. Your character , Mathematical Bold Capital A, has a code point of U+1D400.

In Unicode encodings, you have code units. These are the natural unit of the encoding: 8-bit for UTF-8, 16-bit for UTF-16, and so on. One or more code units encode a single code point.

In UTF-16, two code units that form a single code point are called a surrogate pair. Surrogate pairs are used to encode any code point greater than 16 bits, i.e. U+10000 and up.

This gets a little tricky in .NET, as a .NET Char represents a single UTF-16 code unit, and a .NET String is a collection of code units.

So your code point (U+1D400) can't fit in 16 bits and needs a surrogate pair, meaning your string has two code units in it:

var highUnicodeChar = "";
char a = highUnicodeChar[0]; // code unit 0xD835
char b = highUnicodeChar[1]; // code unit 0xDC00

Meaning when you index into the string like that, you're actually only getting half of the surrogate pair.

You can use IsSurrogatePair to test for a surrogate pair. For instance:

string GetFullCodePointAtIndex(string s, int idx) =>
    s.Substring(idx, char.IsSurrogatePair(s, idx) ? 2 : 1);

Important to note that the rabbit hole of variable encoding in Unicode doesn't end at the code point. A grapheme cluster is the "visible thing" most people when asked would ultimately call a "character". A grapheme cluster is made from one or more code points: a base character, and zero or more combining characters. An example of a combining character is an umlaut or various other decorations/modifiers you might want to add. See this answer for a horrifying example of what combining characters can do.

To test for a combining character, you can use GetUnicodeCategory to check for an enclosing mark, non-spacing mark, or spacing mark.

Perfect! This solution is exactly what I was looking for, and great explanation as well. — hargle, Oct 01 '18 at 06:09
An example of "zero or more combining characters" can be seen on https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags and there are tools to generate this, like https://lingojam.com/GlitchTextGenerator — Ismael Miguel, Oct 01 '18 at 12:37
“code points are 21 bits long” – though it's true that 21 bits is the amount of data with which you can represent any code point, it's not really practically very meaningful to say that this is the “length of a code point”. I don't think such a representation is used anywhere important; for direct access of codepoints you'd actually use UTF-32 or perhaps store the code points in 64 or even 128 bits for reasons of memory uniformity. — Also: umlauts are usually not implemented as combining characters, since most combinations already have a single code point allocated to them. — leftaroundabout, Oct 01 '18 at 15:30
@leftaroundabout when identifying the distinction between Unicode and one of its encoding, I find the concept of "21 bits" to be a good way to break people loose of the sorta-but-not-really correct "UTF32=codepoint" idea. — Cory Nelson, Oct 01 '18 at 16:41

dbc · Answer 2 · 2018-10-01T10:08:17.080

It appears that you want to extract the first "atomic" character from the user point of view (i.e. the first Unicode grapheme cluster) from the highUnicodeChar string, where an "atomic" character includes both halves of a surrogate pair.

You can use StringInfo.GetTextElementEnumerator() to do just this, breaking a string down into atomic chunks then taking the first.

First, define the following extension method:

public static class TextExtensions
{
    public static IEnumerable<string> TextElements(this string s)
    {
        // StringInfo.GetTextElementEnumerator is a .Net 1.1 class that doesn't implement IEnumerable<string>, so convert
        if (s == null)
            yield break;
        var enumerator = StringInfo.GetTextElementEnumerator(s);
        while (enumerator.MoveNext())
            yield return enumerator.GetTextElement();
    }
}

Now, you can do:

var result2 = highUnicodeChar.TextElements().FirstOrDefault() ?? "";

Note that StringInfo.GetTextElementEnumerator() will also group Unicode combining characters, so that the first grapheme cluster of the string Ĥ=T̂+V̂ will be Ĥ not H.

Sample fiddle here.

+1 this is the only reasonable approach. Unfortunately the .NET API (same as most other languages) does not exactly encourage it. There should be a linter for .NET that flags very random access of `char`s inside a `string` as an error. — Konrad Rudolph, Oct 01 '18 at 13:44

Convert Unicode surrogate pair to literal string

2 Answers2