Basic issue regarding full unicode in C++

Question

What are the proper facilities to be using for full unicode in C++?

For example, I have tried:

int main()                                                                                                                                                                 
{                                                                                                                                                                          
    std::wstring name;                                                                                                                                                 
    std::wcout << "Enter unicode: " << std::endl;                                                                                                                
    std::getline(std::wcin, name);                                                                                                                                     

    std::wcout << name << std::endl;                                                                                                                                   

    return 0;                                                                                                                                                              
}  

And it doesn't work as I would expect when entering the character: or others that are not in the Unicode BMP. I get an empty line printed out.

Plain string works for any code points up to 16bits, wstring, wcin, wcout just don't work as I'd expect and some Googling hasn't helped me see what about this could be wrong.

EDIT (file I/O also has issues!):

I wondered if this could have something do with with the console I/O itself and wanted to try the same for file I/O as an experiment. I looked into the api's and came up with this which compiles and runs fine:

int main()                                                                                                                                                                 
{                                                                                                                                                                          
    std::string filename;                                                                                                                                                  
    std::cout << "Enter file to append to: " << std::endl;                                                                                                                 
    std::getline(std::cin, filename);                                                                                                                                      

    std::wifstream file;                                                                                                                                                   
    std::wstringstream buff;                                                                                                                                               
    file.open(filename);                                                                                                                                                   
    std::wstring txt;                                                                                                                                                      
    buff << file.rdbuf();                                                                                                                                                  
    file.close();                                                                                                                                                          
    txt = buff.str();                                                                                                                                                      

    std::wcout << txt << std::endl;                                                                                                                                        

    return 0;                                                                                                                                                              
}                                                                                                                                                                          

But when I point it to my file with mostly lorem ipsum and a few non-BMP characters, it prints the file up to the first non-BMP character and then stops early. Can the Unicode facilities in modern C++ really be this bad?

I'm sure someone knows something basic I am missing here...

Answer 1

You are in the gray zone of C++ unicode. Unicode initially started by an extension of the 7 bits ASCII characters, or multi-byte characters to plain 16 bits characters, what later became the BMP. Those 16 bits characters were adopted natively by languages like Java and systems like Windows. C and C++ being more conservative on a standard point of view decided that wchar_t would be an implementation dependant wide character set that could be 16 or 32 bits wide (or even more...) depending on requirement. The good side was that it was extensible, the dark side was that it was never made clear how non BMP unicode characters should be represented when wchar_t is only 16 bits.

UTF-16 was then created to allow a standard representation of those non BMP characters, with the downside that they need 2 16 bits characters, and that the std::char_traits<wchar_t>::length would again be wrong if some of them are present in a wstring.

That's the reason why most C++ implementation choosed that wchar_t basic IO would only process correctly BMP unicode characters for length to return a true number of characters.

The C++-ish way is to use char32_t based strings when full unicode support is required. In fact wstring_t and wchar_t (prefix L for litteral) are implementation dependant types, and since C++11, you also have char16_t and u16string (prefix u) that explicitely use UTF-16, or char32_t and u32string (prefix U) for full unicode support through UTF-32. The problem of storing characters outside the BMP in a u16string, is that you lose the property size of string == number of characters, which was a key reason for using wide characters instead of multi-byte characters.

One problem for u32string is that the io library still has no direct specialization for 32 bit characters, but as the converters have, you can probably use them easily when you process files with a std::basic_fstream<char32_t> (untested but according to standard should work). But you will have no standard stream for cin, cout and cerr, and will probably have to process the native from in string or u16string, and then convert everything in u32string with the help of the standard converters introduced in C++14, or the hard way if using only C++11.

The really dark side, is that as that native part currently depend on the OS, you will not be able to setup a fully portable way to process full unicode - or at least I know none.