15

In Unicode, letters with accents can be represented in two ways: the accentuated letter itself, and the combination of the bare letter plus the accent. For example, é (+U00E9) and e´ (+U0065 +U0301) are usually displayed in the same way.

R renders the following (version 3.0.2, Mac OS 10.7.5):

> "\u00e9"
[1] "é"
> "\u0065\u0301"
[1] "é"

However, of course:

> "\u00e9" == "\u0065\u0301"
[1] FALSE

Is there a function in R which converts two-unicode-character-letters into their one-character form? In particular, here it would collapse "\u0065\u0301" into "\u00e9".

That would be extremely handy to process large quantities of strings. Plus, the one-character forms can easily be converted to other encodings via iconv -- at least for the usual Latin1 characters -- and is better handled by plot.

Thanks a lot in advance.

AlxH
  • 576
  • 4
  • 14
  • You might want to post your edit as an answer. That way the question shows up as answered. Also, IIRC, you should convert to form D, not C, since the combined characters are a bit of a hack. – krzysz00 Dec 09 '13 at 01:27
  • 1
    Thanks! You might be right about form D in general, though until now form C has seemed more adapted to my practice (e.g. `iconv("\u0065\u0301", to="ASCII//TRANSLIT")` gives `NA`, whereas `iconv(normalize_C("\u0065\u0301"), to="ASCII//TRANSLIT")` gives `"'e"` ; and `plot` prints the labels better in form C). I will try to learn more about the pros and cons. – AlxH Dec 09 '13 at 16:44
  • @AlxH you saved my day - thanks ! – marsei Mar 02 '16 at 17:17

1 Answers1

15

Ok, it appears that a package has been developed to enhance and simplify the string manipulation toolbox in R (finally!). It is called stringi and looks very promising. Its documentation is very well written, and in particular I find the pages about encodings and locales much more enlightening than some of the standard R documentation on the subject.

It has Unicode normalization functions, as I was looking for (here form C):

> stri_trans_nfc('\u00e9') == stri_trans_nfc('\u0065\u0301')
[1] TRUE

It also contains a smart comparison function which integrates these normalization questions and lessens the pain of having to think about them:

> stri_compare('\u00e9', '\u0065\u0301')
[1] 0
# i.e. equal ;
# otherwise it returns 1 or -1, i.e. greater or lesser, in the alphabetic order.

Thanks to the developers, Marek Gągolewski and Bartek Tartanus, and to Kurt Hornik for the info!

gagolews
  • 12,836
  • 2
  • 50
  • 75
AlxH
  • 576
  • 4
  • 14
  • 3
    Since i found your answer when googling for umlaut replacement on german OSX in R via `iconv()`, it might be worthy that with the `stringi` package it simply is one function: `stringi::stri_trans_general(c("äöüø"),"latin-ascii")` – zerweck Jan 12 '16 at 14:12