Problem with getline and "strange characters"

Question

I have a strange problem, I use

wifstream a("a.txt");
wstring line;
while (a.good()) //!a.eof()  not helping
{
     getline (a,line);
      //...
     wcout<<line<<endl;

}

and it works nicely for txt file like this http://www.speedyshare.com/files/29833132/a.txt (sorry for the link, but it is just 80 bytes so it shouldn't be a problem to get it , if i c/p on SO newlines get lost) BUT when I add for example 水 (from http://en.wikipedia.org/wiki/UTF-16/UCS-2#Examples )to any line that is the line where loading stops. I was under the wrong impression that getline that takes wstring as one input and wifstream as other can chew any txt input... Is there any way to read every single line in the file even if it contains funky characters?

For some reason that can only match your user name, wifstream defaults to 8-bit characters. You have to use std::codecvt to tell it to recognize other encodings. — Hans Passant, Aug 12 '11 at 12:54
@ kerrek getline (a,line); fails the same(g++ 4.6, linux). BTW shouldn it get the template param from the arguments? — NoSenseEtAl, Aug 12 '11 at 12:59
I suppose it should. I don't really know, though, if your file is in UTF-16, I don't know if there's a standard C++ way to read this directly (but try what Hans said) -- worst case you can read it in in binary and run it through `iconv()`... remember that "wide string" doesn't imply any particular encoding. It's really more of an internal type than a (de)serializable type. — Kerrek SB, Aug 12 '11 at 13:05
wchar_t has good chances to be 4 bytes on your linux box. Might be the problem ? — Jem, Aug 12 '11 at 13:09

score 8 · Accepted Answer · answered Aug 12 '11 at 13:12

The not-very-satisfying answer is that you need to imbue the input stream with a locale which understands the particular character encoding in question. If you don't know which locale to choose, you can use the empty locale.

For example (untested):

std::wifstream a("a.txt");
std::locale loc("");
a.imbue(loc);

Unfortunately, there is no standard way to determine what locales are available for a given platform, let alone select one based on the character encoding.

The above code puts the locale selection in the hands of the user, and if they set it to something plausible (e.g. en_AU.UTF-8) it might all Just Work.

Failing this, you probably need to resort to third-party libraries such as iconv or ICU.

Also relevant this blog entry (apologies for the self-promotion).

Hi, your solution "worked". It doesnt crash the reading, but I still get ? instad of the real chars when I try to wcout it... imbueing wcout doesnt help(I dont know even if it makes any sense- I was just: "lets try that..."). Now I see why all the hate for the utf 16 and iostreams... :) Still tnx for the nice answ. — NoSenseEtAl, Aug 12 '11 at 13:57

score 4 · Answer 2 · answered Aug 12 '11 at 13:09

4

The problem is with your call to the global function getline (a,line). This takes a std::string. Use the std::wistream::getline method instead of the getline function.

answered Aug 12 '11 at 13:09

David Hammen

32,454
9
60
108

Maybe he's using the getline function because he wants it in a wstring? The getline method on wistream does not output into a wstring. – Alastair Aug 12 '11 at 13:20

score 4 · Answer 3 · edited May 23 '17 at 11:59

C++ fstreams delegeate I/O to their filebufs. filebufs always read "raw bytes" from disk and then use the stream locale's codecvt facet to convert between these raw bytes into their "internal encoding".

A wfstream is a basic_fstream<wchar_t> and thus has a basic_filebuf<wchar_t> which uses the locale's codecvt<wchar_t, char> to convert the bytes read from disk into wchar_ts. If you read a UCS-2 encoded file, the conversion must thus be performed with a codecvt who "knows" that the external encoding is UCS-2. You thus need a locale with such a codecvt (see, for example, this SO question)

By default, the stream's locale is the global locale at the stream construction. To use a specific locale, it should be imbue()-d on the stream.

Problem with getline and "strange characters"

3 Answers3