Writing UTF BOM in C++20

Question

I had the code

#include <string>
#include <fstream>
int main()
{
    std::string BOM = u8"\uFEFF";  // C2440
    std::ofstream f("utf8.txt");
    f << BOM;
}

which worked in C++14 and C++17.

Switching to C++20 now, I get the compiler error C2440. UTF-8 string literals seem to have undergone a breaking change.

The MSDN article suggests a reinterpret_cast, but I can't do that for a string. (And I really doubt that this cast would be a good idea).

std::string BOM = reinterpret_cast<std::string> (u8"\uFEFF");

How do I make my code work without fiddling with the UTF-8 BOM Bytes like 0xEF 0xBB 0xBF? I somehow liked how \uFEFF worked nicely with all sorts of encodings.

I also tried:

#include <string>
#include <fstream>
int main()
{
    std::u8string BOM = u8"\uFEFF";  // ok now
    std::ofstream f("utf8.txt");
    f << BOM;                        // C2679, no matching << operator
}

in which case the u8string line is fine, but the stream output is not.

Not a duplicate, but [see also](https://stackoverflow.com/q/55556200/4137916). UTF-8 support in C++20 seems to be... problematic. — Jeroen Mostert, Nov 10 '22 at 08:19
@JeroenMostert: thanks. It looks like I fell into that trap. — Thomas Weller, Nov 10 '22 at 08:25
I think you have misinterpreted the suggestion. Have you already seen https://stackoverflow.com/questions/56833000/c20-with-u8-char8-t-and-stdstring ? — Bob__, Nov 10 '22 at 09:34
What about not using neither `u8` nor `u8string`? [Demo](https://godbolt.org/z/TMPErefPq) — rturrado, Nov 10 '22 at 09:59
@rturrado: did you try that with a file? It doesn't give a UTF-8 BOM. — Thomas Weller, Nov 10 '22 at 10:07

score 0 · Answer 1 · answered Nov 10 '22 at 10:12

This accepted answer mentioned by @Bob_ will do for my uses.

So my code looks like

#include <string>
#include <fstream>

std::string from_u8string(std::u8string const& s) {
    return { s.begin(), s.end() };
}

int main()
{
    std::string const BOM{ from_u8string(u8"\uFEFF") };
    std::ofstream f("utf8.txt");
    f << BOM << "你好";
}

Writing UTF BOM in C++20

1 Answers1