How to properly Normalize a String with composite characters?

Question

Java Normalize already allows me to take accented characters and output non-accented characters. It does not, however, seem to deal with composite characters (Œ, Æ) very well at all.

Is there a way for Java to deal with these characters natively? I'd like to prevent having to keep a Map of these characters (as that was the reason we moved to using Normalize in the first place).

For example, an input of "Œ" should return "OE", in much the same way it already neatly decomposes characters such as "½" into "1/2".

Please elaborate _It does not, however, seem to deal with composite characters (Œ, Æ) very well at all_ — Sotirios Delimanolis, Jan 22 '18 at 16:02
@SotiriosDelimanolis I think he wants ```Normalizer.normalize("Œ", Normalizer.Form.NFD).equals("OE");``` to be true. Me too. — spi, Jan 22 '18 at 16:09
See the diagram in http://www.unicode.org/reports/tr15/tr15-23.html. It implies that you need Normalizer.Form.NFKD instead. — DodgyCodeException, Jan 22 '18 at 16:29
@DodgyCodeException hum, that seems not be enough (I get empty string as a result too) — spi, Jan 22 '18 at 16:46
@DodgyCodeException I am using NFKD, as that DOES help for the Ǌ composite - but not here. — Weckar E., Jan 23 '18 at 08:49
See the comments following this answer: https://stackoverflow.com/a/7171932/638028 — DodgyCodeException, Jan 23 '18 at 10:44
@DodgyCodeException I don't quite see how that helps? The issue remained unsolved in those comments. — Weckar E., Jan 23 '18 at 16:23
@WeckarE. I know, it helps in the sense that it's telling you it can't be solved ;-) — DodgyCodeException, Jan 23 '18 at 16:25
@DodgyCodeException I choose to believe a solution not having been found yet and no solution existing are two very different things. — Weckar E., Jan 25 '18 at 09:23
Yeah. The fact your question is upvoted so much means that a solution would be highly desirable to many. — DodgyCodeException, Jan 25 '18 at 10:17
Ok, try this: https://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/2013/docs/designDoc/UDF/unicode/NormOperations/splitLigatures.html - especially the bit near the end that mentions icu4j. — DodgyCodeException, Jan 25 '18 at 11:21

Zixradoom · Accepted Answer · 2018-05-14T14:28:41.903

TLDR; No, there is no way with native java to handle these uniformly.

Long Answer

As noted in this question, Separating Unicode ligature characters, the Java Normalizer implementation does not support all of the ligatures that exist in written language.

The reason for this is because Unicode does not support all of the ligatures that exist in written language. Ligatures are a debated subject when it comes to the storage of written language because an argument can be made that they are unimportant from a data viewpoint and that they are important from a layout view point.

The Data viewpoint claims that no information is lost and so it makes more sense to only use the decomposed forms and that the composed forms should not be in Unicode.

The Layout viewpoint claims that the composed ligature represents the proper layout of the written form of language and so should be represented in the data with a special code.

Possible Solution

I would suggest creating a Service that has an interface that handles ligatures only. Supply a concrete implementation that handles all that you currently need. In the future if new implementations are needed it will be simple to add them without modifying the original code by simply adding a new JAR to the program class-path that adds the missing ligatures.

The skeletal implementation may look like this.

Please note I have omitted the code that actually uses the ServiceLoader to locate the LigatureDecoder and LigatureEncoder implementations.

final class Ligatures {
  public static CharSequence compose ( CharSequence decomposedCharacters );
  public static CharSequence decompose ( CharSequence composedCharacters );
}

interface LigatureDecoder {
  CharSequence decompose ( CharSequence composedCharacters );
}

interface LigatureEncoder {
  CharSequence compose ( CharSequence decomposedCharacters );
}

How to properly Normalize a String with composite characters?

1 Answers1