qDebug isn't printing a full QByteArray containing binary data

Question

I have a QByteArray to store data received from a GPS, which is part binary and part ASCII. I want to know for debug proposals know what's being received, so I'm writing a qDebug like this:

//QByteArray buffer;
//...
qDebug() << "GNSS msg (" << buffer.size() << "): " << buffer;

And I get messages like this at console:

GNSS msg ( 1774 ): "ygnnsdgk...(many data)..PR085hlHJGOLH
(more data into a new line, which is OK because it is a new GNSS sentence and
probably has a \n at the end of each one) blablabla...

But suddenly I get a new print iteration. Data has not been erased yet, it has been appended. So new message size its for example 3204, bigger than the previous print obviously. But it prints exactly the same (but with the new size 3204 between brackets). No new data is printed, just the same as the previous message had:

GNSS msg ( 3204 ): "ygnnsdgk...(many data)..PR085hlHJGOLH
(more data into a new line, which is OK because it is a new GNSS sentence and
probably has a \n at the end of each one) blablabla...

I guess qDebug stops printing because it has a limit, or because it reaches a terminating character or something like that, but I'm only guessing.

Any help or explanation for this behaviour?

leemes · Accepted Answer · 2012-06-06T17:55:18.817

Solution / workaround:

Indeed, the qDebug() output of QByteArray gets truncated at a '\0' character. This doesn't have something to do with the QByteArray; you even can't ever output a '\0' character using qDebug(). For an explanation see below.

QByteArray buffer;
buffer.append("hello");
buffer.append('\0');
buffer.append("world");

qDebug() << "GNSS msg (" << buffer.size() << "): " << buffer;

Output:

GNSS msg ( 11 ):  "hello

Even any following arguments are ignored:

qDebug() << "hello" << '\0' << "world";

Output:

hello

You can work around this "problem" by replacing the special characters in your byte array before debugging them:

QByteArray dbg = buffer;   // create a copy to not alter the buffer itself
dbg.replace('\\', "\\\\"); // escape the backslash itself
dbg.replace('\0', "\\0");  // get rid of 0 characters
dbg.replace('"', "\\\"");  // more special characters as you like

qDebug() << "GNSS msg (" << buffer.size() << "): " << dbg; // not dbg.size()!

Output:

GNSS msg ( 11 ):  "hello\0world"

So why is this happening? Why can't I output a '\0' using qDebug()?

Let's dive into the Qt internal code to find out what qDebug() does. The following code snippets are from the Qt 4.8.0 source code.

This method is called when you do qDebug() << buffer:

inline QDebug &operator<<(const QByteArray & t) {
    stream->ts  << '\"' << t << '\"'; return maybeSpace();
}

The stream->ts above is of type QTextStream, which converts the QByteArray into a QString:

QTextStream &QTextStream::operator<<(const QByteArray &array)
{
    Q_D(QTextStream);
    CHECK_VALID_STREAM(*this);
    // Here, Qt constructs a QString from the binary data. Until now,
    // the '\0' and following data is still captured.
    d->putString(QString::fromAscii(array.constData(), array.length()));
    return *this;
}

As you can see, d->putString(QString) is called (the type of d is the internal private class of the text stream), which calls write(QString) after doing some padding for constant-width fields. I skip the code of putString(QString) and directly jump into d->write(QString), which is defined like this:

inline void QTextStreamPrivate::write(const QString &data)
{
    if (string) {
        string->append(data);
    } else {
        writeBuffer += data;
        if (writeBuffer.size() > QTEXTSTREAM_BUFFERSIZE)
            flushWriteBuffer();
    }
}

As you can see, the QTextStreamPrivate has a buffer. This buffer is of type QString. So what happens when the buffer is finally printed on the terminal? For this, we have to find out what happens when your qDebug() statement finishes and the buffer is passed to the message handler, which, per default, prints the buffer on the terminal. This is happening in the destructor of the QDebug class, which is defined as follows:

inline ~QDebug() {
   if (!--stream->ref) {
      if(stream->message_output) {
         QT_TRY {
            qt_message_output(stream->type, stream->buffer.toLocal8Bit().data());
         } QT_CATCH(std::bad_alloc&) { /* We're out of memory - give up. */ }
      }
      delete stream;
   }
}

So here is the non-binary-safe part. Qt takes the textual buffer, converts it to "local 8bit" binary representation (until now, AFAIK we should still have the binary data we want to debug).

But then passes it to the message handler without the additional specification of the length of the binary data. As you should know, it is impossible to find out the length of a C-string which should also be able to hold '\0' characters. (That's why QString::fromAscii() in the code above needs the additional length parameter for binary-safety.)

So if you want to handle the '\0' characters, even writing your own message handler will not solve the problem, as you can't know the length. Sad, but true.

I would not quote the word "problem", it really looks like a bug (as evidenced by the missing closing " after the string which qDebug normally adds. — Tamás Szelei, Jun 06 '12 at 13:05
@fish I also think that just truncating the data at a `'\0'` is not nice. On the other side, I'd guess that qDebug outputs are made for textual data (at least for QString and QByteArray). One might argue that in case of a QByteArray, this *should not* be assumed, but then one can also argue that Qt might not know how you want the binary data to be debugged. Outputting a zero character works but one can't read it (which is the purpose of qDebug()). Maybe a .toHex() is the better solution for binary data in some cases. So I guess Qt leaves it up to you *how* to output the data. — leemes, Jun 06 '12 at 13:12
Apart from that it can be argued that this is a correct behavior, I'm pretty sure it was unintentional, otherwise it would append the missing ". — Tamás Szelei, Jun 06 '12 at 13:15
Is it also possible that when using QByteArray.indexOf("sample"), the string "sample" is never found if placed after the \0 for the same reason? — Roman Rdgz, Jun 06 '12 at 13:17
No. `QByteArray` *is designed* for binary data and thus "binary-safe" (that means that it treats all 256 possible values of a byte equally; none of them is treated specially like in a C-string, where the 0 marks the end). Only the `qDebug` output behaves non-binary-safe. Even the `QDataStream` serialisation, which is used to encode and decode Qt data types to/from binary data stream (to transmit it to another application for example, or save it in a file), is binary-safe. So maybe the Qt developers forgot to care about binary-safety, or it just wasn't a design requirement for the debug output. — leemes, Jun 06 '12 at 16:47
@fish I just realised that the code snippet I quoted above doesn't explain why there is no closing `"` after the truncated byte array. I check the source code again and see what `stream->ts` does when streaming in a `QByteArray` and update my answer. (I just want the *"why"* to be complete.) — leemes, Jun 06 '12 at 16:50
So I updated the answer. It should now be clear what is happening. So it's a problem with the interface of the Qt message handlers, which are designed for textual strings rather than binary data. On one hand, this makes sense, since they are *message* handlers, and *messages* are usually textual, meaning they will not contain a '\0' character. On the other hand, as qDebug() pre- and appends the QByteArray with a `"`, it would be intuitive that the output is somewhat of a C-string reusable in a source code (apart from the '\0'). If the contents of the byte array is `foo", "bar` the output ... — leemes, Jun 06 '12 at 17:20
... will be `"foo", "bar"`, which is even more confusing. In my opinion (and this is what is done by the workaround in the answer), it should either output `"foo\", \"bar"`, or don't output the extra `"`, resulting in an output of `foo", "bar`. No matter how you look at it, debugging a binary byte array (containing non-textual characters) is difficult, and `qDebug()` can't do it. — leemes, Jun 06 '12 at 17:22

qDebug isn't printing a full QByteArray containing binary data

1 Answers1