How to printf a unicode string with '%s' specifier?

Question

I am trying to call printf() to output a Unicode character/string using %s, but it doesn't print anything.

If I call printf() like this:

 printf("\xE2\x98\xA0")

I get a ☠.

But, if I use %ls like this:

printf("%ls", "☠")  /* or */
printf("%ls", L"☠") /* or */
printf("%ls", L"\xE2\x98\xA0")

I get nothing printed;

Also, how can I declare a wchar_t string with Unicode characters inside of it? wchar_t wstro[50] = L"☠" doesn't work.

Do I need to malloc() a wchar_t then put Unicode data in it?

Answer 1

You are confusing Unicode, with UTF-8, and both with wchar_t.

Unicode is something abstract, with code point, combining characters and other properties.

UTF-8 is a common way to encode Unicode, and that it is compatible with ASCII (in case of ASCII only strings), and compatible with C strings (so zero terminated (and no other 0 bytes in the string). \xE2\x98\xA0 is UTF-8 representation.

The character ☠ is probably also encoded in UTF-8. This depends on your editor, but often editors do not use wchar_t.

So: with UTF-8 you should just use %s and not %ls. So your 3 attempts are wrong.

I general, use UTF-8 and so char* and normal strings functions (just do no break strings at random byte, but this mean also not to break strings after random UTF-8 code points if it is followed by some combining code points.

You may use wchar_t, but usually with protocols that use wchar_t, but especially on this case, you should make extra care, because the size of wchar_t could not be compatible with the desired character size (of expected encoding) [e.g. your system and so wchar_t could be just 2 bytes, but so you can use UCS2, but not UTF-32, or the contrary if the system defines wchar_t as 4 bytes).

So keep things simple and try to use just UTF-8, and use it as normal C strings.

Answer 2

This answer assumes you are working in MS Windows

---

It's pretty sad that we're in 2018 and this stuff all still doesn't work properly . But here is the state of things:

printf("\xE2\x98\xA0"); (which is the same as printf("%s", "\xE2\x98\xA0");) works because you are just outputting 3 characters to the output stream. There is no Unicode or special character processing occuring in the C language. It is your terminal environment which looks for UTF-8 strings in the output and chooses display glyphs accordingly.

Similarly, if you wrote the output to a file (using fprintf, or stream redirection) you would see the file contains 0xE2, 0x98, 0xA0 and then you may choose to use a text file viewer that converts UTF-8 to display glyphs.

This part is all fine, and you can (and probably should) write your program to only ever write UTF-8 encoded characters to FILE streams.

---

The problem starts when we want to output wchar_t characters. In theory this should work:

printf("%ls", L"\u2620");   

What is supposed to happen is that wcstombs is called to convert the unicode code point sequence into a multi-byte sequence. But which multi-byte format to use? UTF-8 has become ubiquitous now, but in the past there were also other formats like ShiftJIS, Big-5 etc.

You have to specify the multibyte format by using setlocale. And the details of locales are implementation-defined.

Here's the kicker. There is no C locale supported by Windows for general UTF-8 output. If you try setlocale(LC_CTYPE, ".65001"); it just doesn't work.

You can output certain subsets of Unicode by using a supported locale. For example the MSDN example using Japanese_Japan.932 works, outputting the Unicode input as Shift-JIS. (Not UTF-8).

What's worse is that if you use the Windows API function WideStringToMultiByte, it does accept the "locale" of CP_UTF8. You can use this function to convert L"\u2620"; to a char buffer and printf that, producing UTF-8 output.

But of course you cannot "plug this in" to the FILE stream processing, which only calls wcstombs and not WideStringToMultiByte.

Why didn't they allow ".UTF-8" as a locale for wcstombs? Malicious behaviour? Who knows.

---

The next thing that should work in theory is:

FILE *fp = fopen("a.txt", "w");
fwide(fp, 1);
fwprintf(fp, L"\u2620");

However in actuality, the MS runtime doesn't actually do anything with fwide; it doesn't support wide-oriented streams. The Microsoft implementations of wprintf family actually just output narrow characters, not wide characters, and they use the same wcstombs method that the narrow printf family did.

So, that code doesn't work, and the code from the Japanese wcstombs example, fwprintf(fp, L"\u3603"); (with the .932 CP set) outputs the multibyte sequence instead of the raw wide character.

To write a UTF-16 file via the stdio.h API you actually have no choice but to use narrow characters and treat it like a binary file.

Answer 3

Code that works for Windows

(I use it)

Environment: W7/64 w/ ConEmu console, W10 Terminal or ConEmu, CP always set to 65001.

Compiler : gcc version 11.2.0 (MinGW-W64 x86_64-posix-seh, built by Brecht Sanders)

DOES NOT WORK in W7 default Windows console even w/ CP65001.

#include <stdio.h>
#include <stdlib.h>                     /* malloc */
#include <string.h>                     /* strlen... */
#include <locale.h>
#include <wchar.h>
//
main( int argc , char *argv[])
{
  // .... code here
  printf("%s", "\u25BA");  /* right triangle */
  // ....
}

Result:

C:\Users\gm\C>gets John MARTHA william
►martha
C:\Users\gm\C>

As previously stated by others, locale does not seem to provide any help in making this work on any consoles I tried.

Other syntaxes (L"\u25BA", printf("%ls",...), ...) did NOT bring me the expected result with this gcc compiler under W7 nor W10 on any consoles I tried.