One file lib to conv utf8 (char*) to wchar_t?

Question

I am using libjson which is awesome. The only problem I have is I need to convert an utf8 string (char*) to a wide char string (wchar_t*). I googled and tried 3 different libs and they ALL failed (due to missing headers).

I don't need anything fancy. Just a one way conversion. How do I do this?

Similar question http://stackoverflow.com/a/6155524/1189124 – Marius Balčytis Jul 13 '12 at 08:18 — Marius Balčytis, Jul 13 '12 at 08:18

Ben Stott · Answer 1 · 2011-04-09T11:07:32.630

8

If you're on windows (which, chances are you are, given your need for wchar_t), use MultiByteToWideChar function (declared in windows.h), as so:

int length = MultiByteToWideChar(CP_UTF8, 0, src, src_length, 0, 0);
wchar_t *output_buffer = new wchar_t [length];
MultiByteToWideChar(CP_UTF8, 0, src, src_length, output_buffer, length);

Alternatively, if all you're looking for is a literal multibyte representation of your UTF8 (which is improbable, but possible), use the following (stdlib.h):

wchar_t * output_buffer = new wchar_t [1024];
int length = mbstowcs(output_buffer, src, 1024);
if(length > 1024){
    delete[] output_buffer;
    output_buffer = new wchar_t[length+1];
    mbstowcs(output_buffer, src, length);
}

Hope this helps.

edited Apr 09 '11 at 11:07

answered Apr 09 '11 at 08:10

Ben Stott

2,218
17
23

@Mehrdad: mbstowcs is the CRT function that will eventually boil down to the Win32 API function MultiByteToWideChar after setting up the parameters. – Johann Gerell Apr 09 '11 at 08:30
@Johann: I meant that you'd forgotten the first `s` in *mbstowcs*. – user541686 Apr 09 '11 at 08:31
Your second code sample is badly broken, as can be seen by comparing your use of `mbstowcs()` with a manual page for that function. By any chance, did you intend to use another function name instead of `mbstowcs()`? – Ciaran McHale Apr 09 '11 at 10:53
Yargh, no. That was just off the top of my head. I managed to confuse the ordering of the function arguments. This is why these things shouldn't be off the top of my head.... – Ben Stott Apr 09 '11 at 11:06
@Mehrdad: Then it was just a rhetorical question. Edit the answer instead. – Johann Gerell Apr 09 '11 at 16:02
@Mehrdad, I think he thought you authored the original post. @Johann Gerell, it was **my** post! – Ben Stott Apr 09 '11 at 23:39

score 1 · Answer 2 · answered May 07 '13 at 04:58

the below successfully enables CreateDirectoryW() to write to C:\Users\ПетрКарасев , basically an easier-to-understand wrapper around the MultiByteTyoWideChar mentioned by someone earlier.

std::wstring utf16_from_utf8(const std::string & utf8)
{
    // Special case of empty input string
if (utf8.empty())
    return std::wstring();

// Шаг 1, Get length (in wchar_t's) of resulting UTF-16 string
const int utf16_length = ::MultiByteToWideChar(
    CP_UTF8,            // convert from UTF-8
    0,                  // default flags
    utf8.data(),        // source UTF-8 string
    utf8.length(),      // length (in chars) of source UTF-8 string
    NULL,               // unused - no conversion done in this step
    0                   // request size of destination buffer, in wchar_t's
    );
if (utf16_length == 0)
{
    // Error
    DWORD error = ::GetLastError();
    throw ;
}


// // Шаг 2, Allocate properly sized destination buffer for UTF-16 string
std::wstring utf16;
utf16.resize(utf16_length);

// // Шаг 3, Do the actual conversion from UTF-8 to UTF-16
if ( ! ::MultiByteToWideChar(
    CP_UTF8,            // convert from UTF-8
    0,                  // default flags
    utf8.data(),        // source UTF-8 string
    utf8.length(),      // length (in chars) of source UTF-8 string
    &utf16[0],          // destination buffer
    utf16.length()      // size of destination buffer, in wchar_t's
    ) )
{
    // не работает сука ... 
    DWORD error = ::GetLastError();
    throw;
}

return utf16; // ура!
}

score 0 · Answer 3 · answered Apr 09 '11 at 08:39

Here is a piece of code i wrote. It seems to work well enough. It returns 0 on utf8 error or when the value is > FFFF (which cant be held by a wchar_t)

#include <string>
using namespace std;
wchar_t* utf8_to_wchar(const char*utf8){
    wstring sz;
    wchar_t c;
    auto p=utf8;
    while(*p!=0){
        auto v=(*p);
        if(v>=0){
            c = v;
            sz+=c;
            ++p;
            continue;
        }
        int shiftCount=0;
        if((v&0xE0) == 0xC0){
            shiftCount=1;
            c = v&0x1F;
        }
        else if((v&0xF0) == 0xE0){
            shiftCount=2;
            c = v&0xF;
        }
        else
            return 0;
        ++p;
        while(shiftCount){
            v = *p;
            ++p;
            if((v&0xC0) != 0x80) return 0;
            c<<=6;
            c |= (v&0x3F);
            --shiftCount;
        }
        sz+=c;
    }
    return (wchar_t*)sz.c_str();
}

I can see two problems. First, returning `sz.c_str()` is returning a dangling pointer because the `sz` variable is stack based and will be destructed when the function returns. Second, the size of `wchar_t` is platform dependent. It is 16 bits wide on Windows and OS X, but is 32 bits wide on many UNIX platforms. Because of this, your statement that a wchar_t cannot hold a value larger than 0xFFFF is incorrect. Even on Windows or OS X, the use of surrogate pairs can provide support for UTF-16 rather than (just) UCS-2. — Ciaran McHale, Apr 09 '11 at 10:25

Ciaran McHale · Answer 4 · 2011-04-09T10:45:29.227

The following (untested) code shows how to convert a multibyte string in your current locale into a wide string. So if your current locale is UTF-8, then this will suit your needs.

const char * inputStr = ... // your UTF-8 input
size_t maxSize = strlen(inputStr) + 1;
wchar_t * outputWStr = new wchar_t[maxSize];
size_t result = mbstowcs(outputWStr, inputStr, maxSize);
if (result == -1) {
    cerr << "Invalid multibyte characters in input";
}

You can use setlocale() to set your locale.

One file lib to conv utf8 (char*) to wchar_t?

4 Answers4