Convert Japanese wstring to std::string

Question

Can anyone suggest a good method to convert a Japanese std::wstring to std::string?

I used the below code. Japanese strings are not converting properly on an English OS.

std::string WstringTostring(std::wstring str)
{
    size_t size = 0;
    _locale_t lc = _create_locale(LC_ALL, "ja.JP.utf8");
    errno_t err = _wcstombs_s_l(&size, NULL, 0, &str[0], _TRUNCATE, lc);
    std::string ret = std::string(size, 0);
    err = _wcstombs_s_l(&size, &ret[0], size, &str[0], _TRUNCATE, lc);
    _free_locale(lc);
    ret.resize(size-1);
    return ret;
}

The wstring is "C\\files\\ブ種別.pdf".

The converted string is "C:\\files\\ブ種別.pdf".

Answer 1

It actually looks right to me.

That is the UTF-8-encoded version of your input (which presumably was UTF-16 before conversion), but shown in its ASCII-decoded form due to a mistake somewhere in your toolchain.

You just need to calibrate your file/terminal/display to render text output as if it were UTF-8 (which it is).

---

Also, remember that std::string is just a container of bytes, and does not inherently specify or imply any particular encoding. So your question is rather "how can I convert UTF-16 (containing Japanese characters) into UTF-8 in Windows" or, as it turns out, "how do I configure my terminal to display UTF-8?".

If your display for this string is the Visual Studio locals window (which you suggest is the case with your comment "I observed the value of the "ret" string in local window while debugging") you are out of luck, because VS has no idea what encoding your string is in (nor does it attempt to find out).

For other aspects of Visual Studio, though, such as the console output window, there are various approaches to work around this (example).

Answer 2

EDIT: some things first. Windows has the notion of the ANSI codepage. It's the default codepage of non-Unicode strings that Windows assumes. Every program that uses non-Unicode versions of Windows API, and doesn't specify the codepage explicitly, uses the ANSI codepage.

The ANSI codepage is driven by the "System default locale" setting in Control Panel. As of Windows 10 May 2020, it's under Region/Administrative/Change system locale. It takes admin rights to change that setting.

By default, Windows with the system default locale set to English uses codepage 1252 as the ANSI codepage. That codepage doesn't contain the Japanese characters. So using Japanese in Unicode unaware programs in that situation is hard or impossible.

It looks like the OP wants or has to use a piece of third part C++ code that uses multibyte strings (std::string and/or char*). That doesn't necessarily mean that it's Unicode unaware, but it might. What the OP is trying to do entirely depends on the way that third party library is coded. It might not be possible at all.

---

Looks like your problem is that some piece of third party code expects a file name in ANSI, and uses ANSI functions to open that file. In an English system with the default value of the system locale, Japanese can't be converted to ANSI, because the ANSI codepage (CP1252 in practice) doesn't contain the Japanese characters.

What I think you should do, you should get a short file name instead using GetShortPathNameW, convert that file path to ANSI, and pass that string. Like this:

std::string WstringFilenameTostring(std::wstring str)
{
    wchar_t ShortPath[MAX_PATH+1];
    DWORD dw = GetShortPathNameW(str.c_str(), ShortPath, _countof(ShortPath));

    char AnsiPath[MAX_PATH+1];
    int n = WideCharToMultiByte(CP_ACP, 0, ShortPath, -1, AnsiPath, _countof(AnsiPath), 0, 0);
    return string(AnsiPath);
}

This code is for filenames only. For any other Japanese string, it will return nonsense. In my test, it converted "日本語.txt" to something unreadable but alphanumeric :)