Converting to UTF-8 from ToUnicodeEx()

Question

I get input using GetAsyncKeyState() which I then convert to unicode using ToUnicodeEx():

wchar_t character[1];
ToUnicodeEx(i, scanCode, keyboardState, character, 1, 0, layout);

I can write this to a file using wfstream like so:

wchar_t buffer[128]; // Will not print unicode without these 2 lines
file.rdbuf()->pubsetbuf(buffer, 128);
file.put(0xFEFF); // BOM needed since it's encoded using UCS-2 LE
file << character[0];

When I open this file in Notepad++ it's in UCS-2 LE, when I want it to be in UTF-8 format. I believe ToUnicodeEx() is returning it in UCS-2 LE format, it also only works with wide chars. Is there any way to do this using either fstream or wfstream by somehow converting into UTF-8 first? Thanks!

`ToUnicodeEx` to get UTF-16LE then `WideCharToMultiByte` to get UTF-8 — Richard Critten, Apr 20 '18 at 13:33

score 2 · Answer 1 · answered Apr 20 '18 at 13:33

You might want to use the WideCharToMultiByte function.

For example:

wchar_t buffer[LEN]; // input buffer
char output_buffer[OUT_LEN]; // output buffer where the utf-8 string will be written
int num = WideCharToMultiByte(
    CP_UTF8,
    0,
    buffer,
    number_of_characters_in_buffer, // or -1 if buffer is null-terminated
    output_buffer,
    size_in_bytes_of_output_buffer,
    NULL,
    NULL);

score 2 · Answer 2 · edited Apr 20 '18 at 13:45

2

Windows API generally refers to UTF-16 as unicode which is a little confusing. This means most unicode Win32 function calls operate on or give utf-16 strings.

So ToUnicodeEx returns a utf-16 string.

If you need this as utf 8 you'll need to convert it using WideCharToMultiByte

edited Apr 20 '18 at 13:45

R Sahu

204,454
14
159
270

answered Apr 20 '18 at 13:35

Mike Vine

9,468
25
44

score 0 · Answer 3 · answered Apr 20 '18 at 14:13

Thank you for all the help, I've managed to solve my problem with additional help from a blog post about WideCharToMultiByte() and UTF-8 here.

This function converts wide char arrays to a UTF-8 string:

// Takes in pointer to wide char array and length of the array
std::string ConvertCharacters(const wchar_t* buffer, int len)
{
    int nChars = WideCharToMultiByte(CP_UTF8, 0, buffer, len, NULL, 0, NULL, NULL);

    if (nChars == 0)
    {
        return u8"";
    }

    std::string newBuffer;
    newBuffer.resize(nChars);
    WideCharToMultiByte(CP_UTF8, 0, buffer, len, const_cast<char*>(newBuffer.c_str()), nChars, NULL, NULL);
    return newBuffer;
}

Converting to UTF-8 from ToUnicodeEx()

3 Answers3