11

I'm looking for suggestions regarding unicode aware std::string library replacements. I have a bunch of code that uses std::string, its iterators etc, and would like to now support unicode strings (free or open source implementations preferred, regex capabilities would be great!).

I'm not sure at this point if I require a complete rewrite or if I can get away with dropping in a new string library that supports all of the std::string interfaces. The unicode world seems very complex and I'm just wanting to enable it in my applications not have to learn every single aspect of it.

btw how does the index operator work when it has to pass back a reference to either a 1, 2,3 or 4 structure which could in theory change to either a 1,2,3 or 4 byte structure. if a larger or smaller sized value is passed, does the shifting back and forth of the internal data representation occur insitu?

Zamfir Kerlukson
  • 1,046
  • 1
  • 10
  • 20
  • 1
    There is always std::wstring, which provides a string made of wide chars - however this is not guaranteed to work with unicode (the size is platform-dependent), but it may help. But it would be a simple drop-in replacement for std::string, as it's defined as: `typedef std::basic_string wstring` – icabod May 17 '11 at 08:30
  • 6
    @icabod Using `std::wstring` in itself does not solve the poster's problem, it will still give incorrect results for e.g. `length()`. In fact, `std::string` is enough if all data is stored as UTF8. Also see: http://stackoverflow.com/questions/402283/stdwstring-vs-stdstring. – AVH May 17 '11 at 08:54

5 Answers5

10

You don't need a complete rewrite if you make sure about what your std::string contains. For example, you could assume (and convert inputs to be sure) that your std::string contain UTF8 encoded strings (for those that need localization). Don't forget that std::string is only a container of raw data, it's not associated with an encoding (even in C++0x, it's only a possibility, not a requirement).

Then when you pass text to other libraries that require different encodings, you can use libraries like UTF8CPP to convert to the required encoding (but most of the time such libraries will do it themselves).

That way makes it simple. UTF8 with standard std::string in your code, enabling passing unicode string to everything else (with conversion if necessary).

There have been a lot of discussions about this in the boost community mailing list. Maybe reading it (if you have enough time...) can help you understand other possible solutions.

Klaim
  • 67,274
  • 36
  • 133
  • 188
  • 1
    I've found this to be a good solution. Note, however, that any library written by Microsoft will assume `char`-based strings are "ANSI" encoded instead of UTF-8, so you'll have to write wrappers around those functions. – dan04 May 17 '11 at 11:49
  • Yes, and you'll have to make sure those conversions are done only when needed, not in the rest of the code. – Klaim May 17 '11 at 12:09
7

Depending on your needs, use std::wstring or the larger and more complex (but de facto standard) ICU: http://site.icu-project.org/

Gurgeh
  • 2,130
  • 15
  • 28
6

what unicode encoding do you need? If utf-8 is ok you can have a look at Glib::ustring

Glib::ustring has much the same interface as std::string, but contains Unicode characters encoded as UTF-8.

davka
  • 13,974
  • 11
  • 61
  • 86
  • 4
    Note that Glib::ustring is (should be) very intolerant of illlegal character sequences; moreover if they are passed, e.g. in the constructor, Glib will let things crash by calling `std::string((char*) 0)` instead of throwing a descriptive exception. – sehe May 17 '11 at 08:34
1

I've written my own C++ UTF-8 library, which is a drop-in replacement of std::wstring/string. The data type that is showed to the user is char32_t, but internally the wide characters are all packed into utf8 char's.

The whole thing is quite fast and its performance is best with few unicode codepoints within many ascii codepoints. All operations that are known from std::string are available with this class (except for substring find) and operate on codepoint indices, in contrast to byte indices.

As a bonus of defensive programming, the whole ANSI range of 0-255 can be used without multibytes :)

Hope this helps!

Jakob Riedle
  • 1,969
  • 1
  • 18
  • 21
1

Asking for "a type like std::string, but for Unicode" is like asking for "a type like unsigned, but for primes." std::string is perfectly capable of storing Unicode, in many encodings - the most generally useful being UTF-8.

What you need to replace is your iterators, not your storage type. The iterators should iterate over the codepoints of the string rather than the bytes. That is, ++i should advance one codepoint, and *i should return a codepoint (via uint32_t) rather than a char.

  • 1
    "capable of storing Unicode, in many encodings" Not true. It is capable to store Unicode *only* in UTF-8. If you want to count CESU-8 (bad idea) and UTF-7 (ancient, deprecated), maybe the count goes up to 3. But it is still not "many" – Mihai Nita May 28 '11 at 10:22
  • @Mihai: Also GB 18030 and modified UTF-8. You might argue none of those matter (although when you need modified UTF-8, you generally _need_ it). I'd argue no encodings besides UTF-16 and UTF-32 really matter, so by your count, there aren't "many" encodings for any data type to support. std::string is also fully capable of storing, e.g. subsets of Unicode encoded as 8859-1. The whole point is that "Unicode support" is not the same as "storage format", and std::string provides a very useful storage format. –  May 28 '11 at 15:59