Explanation needed for an UTF-8 vs cpp case

Question

I have Microsoft Visual Studio 2010 on Windows 7 64bit. (In project properties "Character set" is set to "Not set", however every setting leads to same output.)

Source code:

  using namespace std;
  char const charTest[] = "árvíztűrő tükörfúrógép ÁRVÍZTŰRŐ TÜKÖRFÚRÓGÉP\n";
  cout << charTest;
  printf(charTest);
  if(set_codepage()) // SetConsoleOutputCP(CP_UTF8); // *1
    cerr << "DEBUG: set_codepage(): OK" << endl;
  else
    cerr << "DEBUG: set_codepage(): FAIL" << endl;
  cout << charTest;
  printf(charTest);

*1: Including windows.h messes up things, so I'm including it from a separate cpp.

The compiled binary contains the string as correct UTF-8 byte sequence. If I set the console to UTF-8 with chcp 65001 and issue type main.cpp, the string displays correctly.

Test (console set to use Lucida Console font):

D:\dev\user\geometry\Debug>chcp
Active code page: 852

D:\dev\user\geometry\Debug>listProcessing.exe
├írv├şzt┼▒r┼Ĺ t├╝k├Ârf├║r├│g├ęp ├üRV├ŹZT┼░R┼É T├ťK├ľRF├ÜR├ôG├ëP
├írv├şzt┼▒r┼Ĺ t├╝k├Ârf├║r├│g├ęp ├üRV├ŹZT┼░R┼É T├ťK├ľRF├ÜR├ôG├ëP
DEBUG: set_codepage(): OK
��rv��zt��r�� t��k��rf��r��g��p ��RV��ZT��R�� T��K��RF��R��G��P
árvíztűrő tükörfúrógép ÁRVÍZTŰRŐ TÜKÖRFÚRÓGÉP

What is the explanation behind that? Can I somehow ask cout to work as printf?

ATTACHMENT

Many says that Windows console does not support UTF-8 characters at all. I'm a Hungarian guy in Hungary, my Windows is set to English (except date formats, they are set to Hungarian) and Cyrillic letters are still displayed correctly alongside Hungarian letters:

(My default console codepage is CP852)

Answer 1

The differences here is how C++ runtime and C library is handles system locale.

To achieve same result with std::cout you'll can try std::ios::imbue method and std::locale

But main issue with utf-8 and C++ described here

C++03 offers two kinds of string literals. The first kind, contained within double quotes, produces a null-terminated array of type const char. The second kind, defined as L"", produces a null-terminated array of type const wchar_t, where wchar_t is a wide-character. Neither literal type offers support for string literals with UTF-8, UTF-16, or any other kind of Unicode encodings.

So anyway it is all implementation specific and thus non-portable, because non of the standard C++ output streams can understand utf-8.

Answer 2

The the command line does seem to kinda work with UTF-8 for my understanding

1. A font capable of displaying UTF-8 characters
2. Set the correct Code Page in the command line (chcp 65001) not sure if this code page supports the full UTF-8 characters but it seems to be the best available

Check it out here and here

[EDIT] actually 65001 actually is UTF-8 after i checked in PowerShell

PS C:\Users\forcewill> chcp 65001
Active code page: 65001
PS C:\Users\forcewill>  [Console]::OutputEncoding


BodyName          : utf-8
EncodingName      : Unicode (UTF-8)
HeaderName        : utf-8
WebName           : utf-8
WindowsCodePage   : 1200
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
IsSingleByte      : False
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 65001

You can use the PowerShell its much more powerful then the old cmd.exe

Edit:About using cout if we're talking in visual studio the correct answer is here a more tourough explanation can be found here about the best practices within visual studio

Answer 3

On Windows, single-byte strings are usually interpreted as ASCII, or some 256-character codepage. That means you won't get real unicode support.

The short answer is: use wide strings (e.g. L""árvíztűr..." - notice the L) then write to wcout instead of cout. Windows usually interprets wide (2 bytes on Windows) strings as UTF-16 (or at least a close variant), so it will work as intended. On Windows always use wide strings to avoid encoding issues.

Answer 4

First of all windows console does not support UTF-8 (codepage 65001, in order to test this open a UTF-8 encoded file that saved with notepad in console and you will see junk data in the console), so in order to check your output you should redirect it to a file or something like that and check result from there (myapp > test.txt).

second in C/C++ char[] is a sequence of characters that can interpreted anyway that programmer want, but UTF-8 is an special protocol to encode unicode character set, so there is no way (beside C++11) that you write a sequence of characters and those character encoded in UTF8 because I will say char p[3] = "اب", but if compiler want to encode this in UTF-8 it need 5 bytes not 3. so you should use something that understand UTF-8.

I suggest using boost::locale::conv::utf_to_utf with wide string constants. for example

std::string sUTF8 = boost::locale::conv::utf_to_utf(L"árvíztűrő tükörfúrógép ÁRVÍZTŰRŐ TÜKÖRFÚRÓGÉP\n");
std::cout << sUTF8; // or printf( "%s", sUTF8.c_str() );

this will ensure that you have UTF-8 string, but again don't check it with console, since it don't understand UTF-8 at all!!.