Windows Console and Qt Unicode Text

Question

I spent a whole day trying to figure this out with no luck. I looked Everywhere but no luck with working code.

OS: Win XP Sp2 IDE & FRAMEWORK: C++, Qt Creator 2.0.

I am trying to output some unicode (UTF-8) text to the windows console but all I see is gibberish in place of the unicode chars. I know the win console does support unicode (since win 2000)... at least according to Wikipedia and many on the net but I don't see how to make it work with Qt. Most "solutions" I've seen (haven't seen many) use C++ and WInAPI tech... which I can't use because that is not the Qt way. I am using QStrings and Qt!

Code is bellow. I took out all the different things I tried to keep the code simple for the post. Hope someone can get the code to work.

#include <QtCore/QCoreApplication>
#include <QString>
#include <QTextStream>          
#include <QDate>
#include <QFile>
using namespace std;

int main(int argc, char *argv[])
{
    QCoreApplication app(argc, argv);

    QTextStream qin(stdin);         
    QTextStream qout(stdout);       

    //The last 2 chars in QString each need a double slash for an accent.
    QString szqLine = QString::fromUtf8("abc áéüóöú őű");

    //I want this text console output to be in red text color.
    qout << "Bellow are some unicode characters: " << endl; 

    //The Win XP console does not display the unicode chars correctly!!
    //The cosole does not display unicode chars even though it is capable
    //according to wikipedia.  I just don't know how with Qt.
    //I want this text output in white(or default font color, not red.)
    qout << szqLine << endl;

    //Would be nice to get some unicode input from console too.
    qout << "Write some unicode chars like above: " << endl;
    QString szqInput;
    szqInput = QString::fromUtf8(qin.readLine());
    qout << "You wrote: " << endl;
    qout << szqInput << endl;



    return app.exec();
}

Answer 1

Okay, I did some testing with this code. No special setup for the console is required.

#include <QTextStream>

#ifdef Q_OS_WIN32
#include <windows.h>
#include <iostream>
#else
#include <locale.h>
#endif

class ConsoleTextStream: public QTextStream {
  public:
    ConsoleTextStream();
    ConsoleTextStream& operator<<(const QString &string);
};

ConsoleTextStream::ConsoleTextStream():
  QTextStream(stdout, QIODevice::WriteOnly)
{
}

ConsoleTextStream& ConsoleTextStream::operator<<(const QString &string)
{
#ifdef Q_OS_WIN32
  WriteConsoleW(GetStdHandle(STD_OUTPUT_HANDLE),
      string.utf16(), string.size(), NULL, NULL);
#else
  QTextStream::operator<<(string);
#endif
  return *this;
}

class ConsoleInput: public QTextStream {
public:
  ConsoleInput();
  QString readLine();
};

ConsoleInput::ConsoleInput():
  QTextStream(stdin, QIODevice::ReadOnly)
{
}

QString ConsoleInput::readLine()
{
#ifdef Q_OS_WIN32
  const int bufsize = 512;
  wchar_t buf[bufsize];
  DWORD read;
  QString res;
  do {
    ReadConsoleW(GetStdHandle(STD_INPUT_HANDLE),
        buf, bufsize, &read, NULL);
    res += QString::fromWCharArray(buf, read);
  } while (read > 0 && res[res.length() - 1] != '\n');
  // could just do res.truncate(res.length() - 2), but better be safe
  while (res.length() > 0 
         && (res[res.length() - 1] == '\r' || res[res.length() - 1] == '\n'))
    res.truncate(res.length() - 1);
  return res;
#else
  return QTextStream::readLine();
#endif
}

int main()
{
#ifndef Q_OS_WIN32
  setlocale(LC_ALL, "");
#endif
  ConsoleTextStream qout;
  qout << QString::fromUtf8("Текст на иврите: לחם גרוזיני מסורתי הנאפה בתנור לבנים\n");
  qout << QString::fromUtf8("Текст на японском: ※当サイト内コンテンツ・画像・写真データの、転載・転用・加工・無断複製は禁止いたします。\n");
  qout << QString::fromUtf8("Текст на европейском: áéüóöú őű\n");
  qout << flush; // needed on Linux
  ConsoleInput qin;
  QString s = qin.readLine();
  qout << s << endl;
  s = qin.readLine(); // one more time, to ensure we read everything ok
  qout << s << endl;
  return 0;
}

On Windows it prints square boxes for all text except Russian and European. It looks like Lucida Console doesn't have support for Hebrew and Japanese. The funny thing is, when I copy the text from the console to the clipboard and then paste somewhere with Unicode support (e. g. in a browser), it does show up correctly. This proves that Windows actually outputs Unicode, just doesn't display it. Some console font with full Unicode support is needed.

Note that in the example above I have overridden only one operator<<(), but I would need to override them all if I wanted to use them, because they return QTextStream& but aren't virtual, so it is necessary to make them all return ConsoleTextStream&, otherwise something like qout << 1 << someUnicodeString won't work correctly.

I also tested this example on Linux with UTF-8 locale, works perfectly.

Console input with ReadConsoleW() works because the console is configured in so-called line input mode by default, so it waits until the user hits Enter but doesn't wait until enough characters available to fill the buffer, so it does exactly what we want: reads a line provided that the buffer size is enough.

Answer 2

You're making mistakes in both phases - input and output.

Input

You can't write
QString szqLine = QString::fromUtf8("abc áéüóöú őű");
and hope to have a valid Unicode string as the result because this is not guaranteed by the C++ Standard (see SO question C++ source in unicode for details).

You can check you don't have a valid Unicode string using code like this

foreach(QChar ch, szqLine) {
  qout << ch.unicode();
}

If szqLine were a valid Unicode string you would get a list of Unicode code points of characters in the string. In case of your string you get no output.

The proper way to do it is like this

QChar const chars[] = { 'a', 'b', 'c', ' ', 255, 233, 252, 243, 246, 250, ' ', 337, 369};
QString s(&chars[0], sizeof(chars)/sizeof(QChar));

See QString::QString ( const QChar * unicode, int size ), QChar::QChar ( int code ) Qt functions and Full UTF-8 Character Map for Unicode code points of your characters.

Output

Windows console uses one specific code page for input and another one for output (see Console Code Pages) when you use standard input/output mechanisms. This constraints the set of characters you can enter and display to these present in the current code page. However you can use WriteConsole Win API function to output any Unicode string encoded in UTF-16. There's no way you can avoid using Win API function here because there's no Qt API that could be used here. Below is complete example showing how to display characters from your question on the Windows console.

#include <QtCore/QCoreApplication>
#include <QString>
#include <QTextCodec>

#include <Windows.h>

using namespace std;

int main(int argc, char *argv[])
{
    QCoreApplication app(argc, argv);

    QChar const chars[] = { 'a', 'b', 'c', ' ', 255, 233, 252, 243, 246, 250, ' ', 337, 369};                
    QString s(&chars[0], sizeof(chars)/sizeof(QChar));

    WriteConsoleW(GetStdHandle(STD_OUTPUT_HANDLE), s.utf16().constData(), s.size(), NULL, NULL);

    return app.exec();
}

Answer 3

I think it's because your code needs to use WriteConsoleW instead of WriteFile internally, and the runtime library might not use that function. If it doesn't use WriteFileW, then you can't write Unicode.

Answer 4

I got a little farther today with the code bellow. Now it displays unicode correctly to the console but still not working quite right because the console freezes/crashes after the first unicode text is put out to the console and nothing subsequent to that is show on the console. It is as if unicode chars cause the console buffer to get confused after the first text output.

#include <QtCore/QCoreApplication>
#include <QString>
#include <QTextStream>
using namespace std;

int main(int argc, char *argv[])
{
    QCoreApplication app(argc, argv);

    QTextStream qin(stdin);
    qin.setCodec("UTF-8");
    qin.autoDetectUnicode();

    QTextStream qout(stdout);
    qout.setCodec("UTF-8");
    qout.autoDetectUnicode();

    //The last 2 chars in QString each need a double slash for an accent.
    QString szqLine = QString::fromUtf8("abc áéüóöú őű");

    qout << "Bellow are some unicode characters: " << endl;

    //Now this is displayed correctly on cosole but after displaying text
    //it no loger is capable of displaying anything subsequently.
    qout << szqLine << endl;

    //Would be nice to get some unicode input from console too.
    qout << "Write some unicode chars like above: " << endl;
    QString szqInput;
    szqInput = qin.readLine();
    qout << "You wrote: " << endl;
    qout << szqInput << endl;

    return app.exec();
}