How to properly convert USC-2 little endian into UTF-8?

Question

I have a file and the line endings are in the windows style \r\n; it is encoded in USC-2 little endian.

Say this is my file fruit.txt (USC-2 little endian):

So I open it in a std::wifstream and try to parse the contents:

// open the file
    std::wifstream file("fruit.txt");
    if( ! file.is_open() ) throw std::runtime_error(std::strerror(errno));

// create container for the lines
    std::forward_list<std::string> lines;

// Add each line to the container
    std::wstring line;
    while(std::getline(file,line)) lines.emplace_front(wstring_to_string(line));

If I try to print to cout...

// Printing to cout
    for( auto it = lines.cbegin(); it != lines.cend(); ++it )
        std::cout << *it << std::endl;

...This is what it outputs:

Cherry
Banana
ÿþApple

Worse yet, if I open it in Notepad++, this is what it looks like

I can sort-of rectify this by forcibly converting the encoding back to USC-2 which results in this:

My wstring_to_string function is defined as this:

std::string wstring_to_string( const std::wstring& wstr ) {
    std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> convert;
    return convert.to_bytes(wstr);
}

What in the world is going on here? How can I get a normal UTF-8 string? I have tried this method too: How to read utf-16 file into utf-8 std::string line by line, but imbuing the std::wifstream first results in no outputs altogether. Can someone please help direct me in the best way to go about converting USC-2 LE data to readable UTF-8 data?

Edit I think there may be a bug with mingw64/mingw-w64-x86_64-gcc 6.3.0-2 which is provided by MSYS2. I have tried everyone's suggestions and imbuing the locale into the streams is just rendering no output at all. I do know there are only two native locales provided, "C" and "POSIX". I was going to try Visual Studio but don't have sufficient internet speed for the 4GB download. I have used ICU like @Andrei R. suggested and it is working great.

I would have loved to use standard libraries but I am ok with this. Please take a look at my code if you need this solution: https://pastebin.com/qudy7yva

Answer 1

The code itself is fine as-is.

The real problem is that your input file is NOT valid UTF-16LE to begin with (your use of std::codecvt_utf8_utf16 requires UTF-16, not UCS-2). This is clearly shown in your Notepad++ screenshots.

Offhand, the file data looks like a UTF-16LE file with a BOM (ÿþ is the UTF-16LE BOM when viewed as 8bit ANSI) was appended as-is to the end of a UCS-2BE (or UTF-16BE) file that did not have a BOM.

You need to fix the input file so the entire file is valid UTF-16LE from beginning to end (with or without a BOM in front, not in the middle).

Then the code you already have will work.

Answer 2

converting to/from unicode is in general not so trivial. Have a look at ICU libraries, I believe, this is by far most complete encoding conversion library for c/c++.

There are also platform-dependent ways like WideCharToMultibyte (Win) or iconv (Linux). Or, with Qt, you can use QString::fromUtf16. Probably you will have to reverse endianness yourself.

Answer 3

For your case, the main issue is that you made the wifstream read the file in a wrong way. If you print the size of wstr in wstring_to_string, you will find that it's not what you expect.

https://stackoverflow.com/a/19698449/4005852

Set proper locale will fix this issue.

std::string wstring_to_string( const std::wstring& wstr ) {
    std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> convert;
    return convert.to_bytes(wstr);
}

int main()
{
// open the file
    std::wifstream file("fruit.txt", std::ios::binary);
    file.imbue(std::locale(file.getloc(),
          new std::codecvt_utf16<wchar_t, 0x10ffff, std::little_endian>));
    if( ! file.is_open() ) throw std::runtime_error(std::strerror(errno));

// create container for the lines
    std::forward_list<std::string> lines;

// Add each line to the container
    std::wstring line;
    file.get(); // remove BOM
    while(std::getline(file,line)) lines.emplace_front(wstring_to_string(line));

// Printing to cout
    for( auto it = lines.cbegin(); it != lines.cend(); ++it )
        std::cout << *it << std::endl;

    return 0;
}