UTF-8 to wide char conversion

Question

#ifndef UNICODE
#define UNICODE
#endif

#include <Windows.h>
#include <cstdio>
#include <fstream>

using namespace std;

int main()
{
    FILE* resFile;
    char multiByteStr[256];
    ifstream oFile;
    FILE* exampleFile;
    TCHAR buffer[256];      
    system("chcp 65001");

    resFile = _wfopen(L"foo",L"w, ccs=UTF-8");             
    fwprintf(resFile,L"%s",L"C:\\exsistingFolder\\zażółć gęśłą jaźń ☺☻♥♦• ć.txt");    
    fclose(resFile);   

    oFile.open(L"foo");    
    oFile.getline(multiByteStr,256,'\n');       
    oFile.close();    

    MultiByteToWideChar(CP_UTF8,0,multiByteStr,256,buffer,256);    
    wprintf(L"%s",buffer);

    exampleFile = _wfopen(buffer,L"w, ccs=UTF-16LE");       
    fwprintf(exampleFile,L"%s",buffer);     
    fclose(exampleFile);

    system("pause");
    return 0;
}

As you can see, program should create file "foo" resFile that contains a full path of the file to be created, and this new file exampleFile should contain a path to itself. Although during debugging in Visual studio 2010 autos yields that buffer has the correct string, exampleFile isn't created. Why?
And another thing: why wprintf doesn't output extended characters, though I've switched console's font to Lucida Console - that one which can deal with uncode characters.

Ps. exampleFile points to NULL, even after _wfopen, and the last character of buffer is '/0'.

Is `buffer` NULL terminated? Did you check the return values of the calls after `oFile.close`? — dirkgently, May 27 '12 at 14:34
possible duplicate of [Making wide char file](http://stackoverflow.com/questions/10769023/making-wide-char-file) — Hans Passant, May 27 '12 at 16:23
To print Unicode to the console, you need `_setmode` in addition to `wprintf` (but no `chcp`) – see http://blogs.msdn.com/b/michkap/archive/2008/03/18/8306597.aspx. — Philipp, May 27 '12 at 17:44

score 2 · Answer 1 · answered May 27 '12 at 17:04

You aren't doing any error handling. Most importantly, when calling MultiByteToWideChar(), you are telling it to convert the entire multiByteStr buffer, but most of the buffer contains garbage because you did not zero it beforehand. You have to use the actual number of characters in the buffer, not the buffer's max size. MultiByteToWideChar() is likely returning an error that you are ignoring. There are several other failure spots in the code as well. You need to always check for errors, especially when interacting with the OS.

Try this:

#define UNICODE

#include <Windows.h>
#include <cstdio>
#include <fstream>

using namespace std;

void pause()
{
    wcin.ignore();
    wcin.get();
}

int main()
{
    FILE* resFile;
    char multiByteStr[256] = {0};
    ifstream oFile;
    FILE* exampleFile;
    WCHAR buffer[256] = {0};

    SetConsoleOutputCP(CP_UTF8);

    resFile = _wfopen(L"foo",L"w, ccs=UTF-8");
    if (!resFile) {
        wprintf(L"Unable to create foo");
        goto done;
    }

    fwprintf(resFile,L"%s",L"C:\\exsistingFolder\\zażółć gęśłą jaźń ☺☻♥♦• ć.txt");
    fclose(resFile);

    if (!oFile.open(L"foo")) {
        wprintf(L"Unable to open foo");
        goto done;
    }

    oFile.getline(multiByteStr,255,'\n');
    oFile.close();

    if (MultiByteToWideChar(CP_UTF8,0,multiByteStr,-1,buffer,256) == 0) {
        wprintf(L"Unable to convert UTF-8 to UTF-16. Error: %u", GetLastError());
        goto done;
    }

    exampleFile = _wfopen(buffer,L"w, ccs=UTF-16LE"); 
    if (!exampleFile) {
        wprintf(L"Unable to create file: %s", buffer);
        goto done;
    }

    fwprintf(exampleFile,L"%s",buffer); 
    fclose(exampleFile);

    wprintf(L"created file: %s", buffer);

done:
    pause();
    return 0;
}

Error-handling had been removed from my code on the purpose of simplicity. You helped me a little with MultiByteToWideChar function, but it didn't resolved my problem. — 0x6B6F77616C74, May 29 '12 at 15:35

score 1 · Answer 2 · answered May 28 '12 at 01:00

1

Why mix and match your I/O libraries? Here's a pure (Windows) C solution, minus error checking:

#include <stdio.h>
#include <stdlib.h>

int main()
{
    FILE* pFile;
    wchar_t buffer[256];

    _wfopen_s(&pFile,L"foo",L"w, ccs=UTF-8");
    fputws(L"C:/existingFolder/zażółć gęśłą jaźń ☺☻♥♦• ć.txt",pFile);
    fclose(pFile);

    _wfopen_s(&pFile,L"foo",L"r, ccs=UTF-8");
    fgetws(buffer,_countof(buffer),pFile);
    fclose(pFile);

    _wfopen_s(&pFile,buffer,L"w, ccs=UTF-16LE");
    fputws(buffer,pFile);
    fclose(pFile);

    return 0;
}

answered May 28 '12 at 01:00

Mark Tolonen

166,664
26
169
251

Have you ever tried to run this? It throws run-time "debug assertion failed" message box , like my program. – 0x6B6F77616C74 May 29 '12 at 15:16
It runs fine, as long as `C:/existingFolder` exists. Add error checking to your version. If the file doesn't open it will crash since the file handle will be NULL. It is also important that you save the source file in an appropriate format. I used UTF-16LE (with BOM). – Mark Tolonen May 29 '12 at 15:45
My mistake, thanks about this , it's useful, it works, but it's still not an answer to my question. – 0x6B6F77616C74 May 29 '12 at 18:17

score 1 · Accepted Answer · answered May 29 '12 at 18:21

1

The solution is very trivial- _wfopen creates a file encoded in UTF-8 with BOM, and MultiByteToWideChar function doesn't remove the BOM, so we need to manually get rid of this.

answered May 29 '12 at 18:21

0x6B6F77616C74

2,559
7
38
65

UTF-8 to wide char conversion

3 Answers3

Linked