9

This is my sample code:

#pragma execution_character_set("utf-8")

#include <boost/locale.hpp>
#include <boost/algorithm/string/case_conv.hpp>
#include <iostream>

int main()
{
    std::locale loc = boost::locale::generator().generate("");
    std::locale::global(loc);

#ifdef MSVC
    std::cout << boost::locale::conv::from_utf("grüßen vs ", "ISO8859-15");
    std::cout << boost::locale::conv::from_utf(boost::locale::to_upper("grüßen"), "ISO8859-15") << std::endl;
    std::cout << boost::locale::conv::from_utf(boost::locale::fold_case("grüßen"), "ISO8859-15") << std::endl;
    std::cout << boost::locale::conv::from_utf(boost::locale::normalize("grüßen", boost::locale::norm_nfd), "ISO8859-15") << std::endl;
#else
    std::cout << "grüßen vs ";
    std::cout << boost::locale::to_upper("grüßen") << std::endl;
    std::cout << boost::locale::fold_case("grüßen") << std::endl;
    std::cout << boost::locale::normalize("grüßen", boost::locale::norm_nfd) << std::endl;
#endif

    return 0;
}

Output on Windows 7 is:

grüßen vs GRÜßEN
grüßen
grußen

Output on Linux (openSuSE 12.3) is:

grüßen vs GRÜSSEN
grüssen
grüßen

On Linux the german letter 'ß' is converted to 'SS' as predicted, while this character remains unchanged on Windows.

Question: why is this so? How can I correct the conversion?

Some notes: Windows console codepage is set to 1252. In both cases locales are set to de_DE. I tried to replace the default locale setting in the listing above by "de_DE.UTF-8" - without any effect. On Windows this code is compiled with Visual Studio 2013, on Linux with GCC 4.7, c++11 enabled.

Any suggestions are appreciated - thanks in advance for your support!

Rainer
  • 111
  • 1
  • 6
  • For starters: you can't mix `cout` and `wcout`. Once you commit, stick to one flavour – sehe Mar 11 '14 at 19:44
  • Following your advice I modified my listing - output is unchanged on both systems. I can live with the problem that 'ß' is not converted to 'SS', but I need to achieve the same behaviour on both systems. Do you have an idea, how this could be done? – Rainer Mar 12 '14 at 07:57
  • Hi, my problem could sound silly, but I am not able to run any example that involves boost.locale in Visual Studio 2010.http://stackoverflow.com/questions/29185984/boost-locale-examples-doesnt-runs-in-vs2010 is the link to my problem. How did you run the code in Visual Studio.? – Mahadeva Mar 28 '15 at 14:34

1 Answers1

3

Windows doesn't do this conversion because "it would be too confusing" for developers if the string length changed all of a sudden. And boost presumably just delegates all the Unicode conversions to the underlying Windows APIs

Source

I guess the robust way to handle it would be to use a third-party Unicode library such as ICU.

jalf
  • 243,077
  • 51
  • 345
  • 550
  • 1
    FWIW, Boost.Locale can use ICU as a backend. – R. Martinho Fernandes Mar 12 '14 at 09:27
  • @jalf and Martinho: "_By default, Boost.Locale uses ICU for all localization and text manipulation tasks_" is quoted from Boost.Locale documentation. On both machines boost uses ICU. I added fold and normalize functions to the listing above - also here I get different output on Windows and Linux. – Rainer Mar 12 '14 at 12:36
  • @Rainer that depends on whether Boost is built with ICU support, and as far as I can tell, even if it is, ICU needs to be installed as well. It doesn't look like it is statically linked into Boost.Locale. Are you sure that your code actually uses the ICU backend? – jalf Mar 12 '14 at 12:45
  • @jalf: on Windows it is build with ICU support - I did it myself and checked the build log. Without linking icuucd.lib the above listed code wouldn't compile. On Linux I used boost from the standard Repo - here I checked the dependencies and as far as I can see at least two libraries from ICU are required for boost locale. – Rainer Mar 12 '14 at 12:58
  • 1
    I fixed it - for anyone, who might step into the same trouble here is the solution: Although I didn't get any error message while compiling nor when the application was executed it was not sufficient to just include icuucd.lib for linking you have to include icuind.lib as well. Then you get the same result on Windows and Linux. – Rainer Mar 15 '14 at 09:31
  • Ah, so it (silently) fell back to the WinAPI backend, I'm guessing. Interesting. Thanks for the update. :) – jalf Mar 16 '14 at 10:36
  • @Rainer i'm getting the same issues. Did you mean you had to link icu in your application and not just while building boost? – Captain Jack sparrow May 16 '20 at 01:12