C++ & Boost: encode/decode UTF-8

Question

I'm trying to do a very simple task: take a unicode-aware wstring and convert it to a string, encoded as UTF8 bytes, and then the opposite way around: take a string containing UTF8 bytes and convert it to unicode-aware wstring.

The problem is, I need it cross-platform and I need it work with Boost... and I just can't seem to figure a way to make it work. I've been toying with

Trying to convert the code to use stringstream/wstringstream instead of files of whatever, but nothing seems to work.

For instance, in Python it would look like so:

>>> u"שלום"
u'\u05e9\u05dc\u05d5\u05dd'
>>> u"שלום".encode("utf8")
'\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d'
>>> '\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d'.decode("utf8")
u'\u05e9\u05dc\u05d5\u05dd'

What I'm ultimately after is this:

wchar_t uchars[] = {0x5e9, 0x5dc, 0x5d5, 0x5dd, 0};
wstring ws(uchars);
string s = encode_utf8(ws); 
// s now holds "\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d"
wstring ws2 = decode_utf8(s);
// ws2 now holds {0x5e9, 0x5dc, 0x5d5, 0x5dd}

I really don't want to add another dependency on the ICU or something in that spirit... but to my understanding, it should be possible with Boost.

Some sample code would greatly be appreciated! Thanks

Does `imbue` not work with `stringstream`? What exactly is going wrong with the utf8 codecvt facet? — Ben Voigt, May 26 '11 at 14:37
Have a look at http://stackoverflow.com/questions/148403/utf8-to-from-wide-char-conversion-in-stl — Mark Ransom, May 26 '11 at 14:50
`wchar_t`/`wstring` is a bad choice for holding codepoints, as there is no guarantee at all that wchar_t is wide enough for that (iirc, on windows it isn't for codepoints outside the BMP. — etarion, May 26 '11 at 16:00

score 25 · Accepted Answer · answered May 27 '11 at 17:07

Thanks everyone, but ultimately I resorted to http://utfcpp.sourceforge.net/ -- it's a header-only library that's very lightweight and easy to use. I'm sharing a demo code here, should anyone find it useful:

inline void decode_utf8(const std::string& bytes, std::wstring& wstr)
{
    utf8::utf8to32(bytes.begin(), bytes.end(), std::back_inserter(wstr));
}
inline void encode_utf8(const std::wstring& wstr, std::string& bytes)
{
    utf8::utf32to8(wstr.begin(), wstr.end(), std::back_inserter(bytes));
}

Usage:

wstring ws(L"\u05e9\u05dc\u05d5\u05dd");
string s;
encode_utf8(ws, s);

score 18 · Answer 2 · answered May 26 '11 at 15:00

18

There's already a boost link in the comments, but in the almost-standard C++0x, there is wstring_convert that does this

#include <iostream>
#include <string>
#include <locale>
#include <codecvt>
int main()
{
    wchar_t uchars[] = {0x5e9, 0x5dc, 0x5d5, 0x5dd, 0};
    std::wstring_convert<std::codecvt_utf8<wchar_t>> conv;
    std::string s = conv.to_bytes(uchars);
    std::wstring ws2 = conv.from_bytes(s);
    std::cout << std::boolalpha
              << (s == "\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d" ) << '\n'
              << (ws2 == uchars ) << '\n';
}

output when compiled with MS Visual Studio 2010 EE SP1 or with CLang++ 2.9

true 
true

answered May 26 '11 at 15:00

Cubbi

46,567
13
103
169

Is it still not cross-platform? After C++11 has arrived? – Mikhail Jul 28 '14 at 10:19
1

@Mikhail it arrived, but GCC did not implement it yet. – Cubbi Jul 28 '14 at 13:15
1

To keep the answer updated, so far `std::codecvt_utf8`, `std::wstring_convert` and family are [deprecated in C++17](http://en.cppreference.com/w/cpp/locale/wstring_convert). You can use [`std::codecvt`](http://en.cppreference.com/w/cpp/locale/codecvt) instead. – cbuchart Jul 06 '17 at 13:26
2

@cbuchart not really. They were deprecated with no alternative, in an attempt to encourage an alternative for C++20. Much like how std::strstream was deprecated in C++98 and is still in the standard with no alternative. – Cubbi Jul 06 '17 at 13:49

Diaa Sami · Answer 3 · 2012-10-09T10:09:15.610

13

Boost.Locale was released in Boost 1.48(November 15th, 2011) making it easier to convert from and to UTF8/16

Here are some convenient examples from the docs:

string utf8_string = to_utf<char>(latin1_string,"Latin1");
wstring wide_string = to_utf<wchar_t>(latin1_string,"Latin1");
string latin1_string = from_utf(wide_string,"Latin1");
string utf8_string2 = utf_to_utf<char>(wide_string);

Almost as easy as Python encoding/decoding :)

Note that Boost.Locale is not a header-only library.

edited Oct 09 '12 at 10:09

answered Oct 08 '12 at 14:23

Diaa Sami

3,249
24
31

Can you add example of decode_utf8 operation with this library ? – Dfr Jul 18 '13 at 13:23
You can try this form to decode utf-8 into utf-16 wstring wide_string = to_utf(utf8_bytes,"utf-8"); – Diaa Sami Jul 20 '13 at 23:34

Jakob Riedle · Answer 4 · 2018-04-09T08:12:11.180

2

For a drop-in replacement for std::string/std::wstring that handles utf8, see TINYUTF8.

In combination with <codecvt> you can convert pretty much from/to every encoding from/to utf8, which you then handle through the above library.

edited Apr 09 '18 at 08:12

answered Nov 30 '16 at 15:37

Jakob Riedle

1,969
1
18
21

1

Hi @jakob-riedle! I have a minor question about your library (BTW, appreciate the elegance and the amount of work you've put in it!). Can you please take a look: https://stackoverflow.com/questions/49716774/tiny-utf8-getting-offset-in-characters-codepoints ? – Vadim Berman Apr 08 '18 at 11:27

C++ & Boost: encode/decode UTF-8

4 Answers4

Linked

Related