9

I have a problem, I need to use UTF-8 encoded strings on standard char types in C++ source code like so:

char* twochars = "\xe6\x97\xa5\xd1\x88";

Normally, if I want to write an UTF-8 character I need to use octets like above. Is there something in Visual Studio (I'm using VS 2013 Ultimate) that could allow me to just write for example "ĄĘĆŻ" and automagically converted each character to multiple UTF-8 octets like in the example above? Or should I use const wchar_t*and find a lib that could convert wide strings to UTF-8 encoded standard char strings?

If there is no such thing, could you suggest any external software for that? I really don't feel like browsing the character map for every symbol/non-latin letter.

Sorry for my English, Thanks in advance.

Michael K. Sondej
  • 963
  • 8
  • 24

2 Answers2

16

You can use the still undocumented pragma directive execution_character_set("utf-8"). This way your char strings will be saved as UTF-8 in your binary. BTW, this pragma is available in Visual C++ compilers only.

#include <iostream>
#include <cstring>

#pragma execution_character_set("utf-8")

using namespace std;

char *five_chars = "ĄĘĆŻ!";

int _tmain(int argc, _TCHAR* argv[])
{
    cout << "This is an UTF-8 string: " << five_chars << endl;
    cout << "...it's 5 characters long" << endl;
    cout << "...but it's " << strlen(five_chars) << " bytes long" << endl;
    return 0;
}
Jigsore
  • 779
  • 4
  • 5
  • 1
    awesome! this was the hint I was looking for minutes. – fmuecke Jul 21 '15 at 20:46
  • Isn't recognised in VS2102. The pragma was supported in VC++ 2010, then forgotten in VC++ 2012, and is supported again in VC++ 2013 [link](http://stackoverflow.com/questions/23607830/does-vc-have-a-compile-option-like-fexec-charset-in-gcc-to-set-the-execution) – TinyRacoon May 17 '16 at 13:41
  • The `execution_character_set` pragma [is finally documented](https://msdn.microsoft.com/en-us/library/mt708823.aspx) as of VS 2015; available also through the compiler option [`/execution-charset`](https://msdn.microsoft.com/en-us/library/mt708818.aspx). Also the new almighty [`/utf-8`](https://msdn.microsoft.com/en-us/library/mt708821.aspx) enables UTF-8 for source files AND execution charset both in one go. – Max Truxa Jan 17 '17 at 15:45
2

There's no way to write the string literal directly in UTF-8 with the current versions of VC++. A future version should have UTF-8 string literals.

I tried pasting non-ASCII text directly into a string literal in a source file and saved the file as UTF-8. Looking at the source file in a hex editor confirmed that it's saved as UTF-8, but that still doesn't do what you want. At compile time, those bytes are either mapped to a character in the current code page or you get a warning.

So the most portable way to create a string literal right now is to explicitly write the octets as you've been doing.

If you want to do a run-time conversion, there are a couple options.

  1. The Windows API has WideCharToMultiByte, which can take a text as UTF-16 and convert it to multibyte encodings like UTF-8.
  2. If you're using a new enough version of the compiler and the C++ runtime, you can use std::codecvt to transform your wide character string into UTF-8.

You could use one of these techniques to write a little utility that does the conversion and outputs them as the explicit octets you would need for a string literal. You could then copy and paste the output into your source code.

Adrian McCarthy
  • 45,555
  • 16
  • 123
  • 175