3

QtWebEngine uses a IPC mechanism to communicate between the C+ Qt world and the JavaScript work. This mechanism is used for QWebChannel, and it appears to be based on WebSockets. Is there a way to use the underlying IPC or WebSockets without using QWebChannel, as the latter seems restricted to strings or JSON-encoded data?

Background: I wrote an application QtDomTerm which is a JavaScript-based terminal emulator that uses QWebChannel to connect input/output from a PTY to QtWebEngine. This works fairly well, but there is a glitch relating to utf8/string conversion. Ideally, I'd like to send raw bytes from the PTY, and do byte-to-text conversion in JavaScript. But QWebChannel is too high-level and only handles strings or JSON-encoded data. It does not handle QByteArray.

Of course there are multiple ways to solve my problem. One is to manually create a WebSocket server, and have the JavaScript running in the QtWebEngine connect to it. But it appears that is what is going on behind the scene anyway, using qt.webChannelTransport. It seems like it would be most efficient and elegant if I could access the underlying transport (the class WebChannelIPCTransportHost seems to be relevant).

Anyone tried something like this? I.e. I would like to not use QWebChannel - unless there is an efficient way for it to pass a QByteArray.

(I rephrased the question. There was a comment about missing research, but I've browsed heavily though the Qt docuemntation, source code, and here, without finding a clear answer.)

Per Bothner
  • 51
  • 4
  • where does it say that QWebChannel is restricted to strings of JSON-encoded data. I recently ran into a problem - whereby my custom Struct - even when put in a QVariant was unable to be sent over a QWebChannel to a JavaScript script. It would be great if I could find the QT reference for this - its not documented in the QWebChannel. – johnco3 Oct 27 '18 at 20:11
  • https://doc.qt.io/qt-5.11/qtwebchannel-javascript.html says send() "takes a stringified JSON message". The https://doc.qt.io/qt-5.11/qwebchannelabstracttransport.html class send and receives QJsonObjects. – Per Bothner Oct 29 '18 at 18:45

2 Answers2

1

What prevents you from sending QString::fromLatin1(data.toHex()), where data is of the QByteArray type? That's all you need, really. Use a reverse conversion on the javascript side, see e.g. this question.

Community
  • 1
  • 1
Kuba hasn't forgotten Monica
  • 95,931
  • 16
  • 151
  • 313
  • Yes, it is certainly doable (and relatively simple) to convert the bytes to hex, and transmitting the resulting string. But it is undeniably inefficient, as it doubles the amount of data transmitted. It also offends my sense of elegance: why convert bytes to hex - when we're using a byte protocol anyway. In my application I really don't need the functionality of QWebChanell - I just need a simple bi-directions byte channel, as I already have a mechanism for "out-of-band" messages. – Per Bothner Sep 26 '16 at 19:10
  • Can't you use a regular TCP socket and websockets? – Kuba hasn't forgotten Monica Sep 26 '16 at 19:23
  • 1
    Using "TCP socket and websockets" is fine (and essentially what I want to do) - but rather than creating a new listener and connection, I think it would be simpler and more robust to piggy-back on the existing IPC connection. (Not all environments have access to TCP sockets, for example.) QWebChannel already sets up a connection; I would just prefer to use the existing connection at a low (byte-stream) level, rather than the higher QWebChannel level. (I updated the question to make that hopefully clearer.) – Per Bothner Sep 26 '16 at 23:06
  • You can use Qt's private implementation details to do it. If you're asking whether the public API supports it: it doesn't. Since you are compiling your own copy of Qt, this shouldn't be a problem. You can even patch your copy to add the lower-level functionality in. You're keeping Qt in your source control, as it's an integral part of your product. Right? Right? :) – Kuba hasn't forgotten Monica Sep 26 '16 at 23:55
  • I am currently using unmodified Qt as installed on Fedora. Even if I were compiling it, making and maintaining a private modification would be much uglier and undesirable than alternative solutions, even if they are uglier than I'd like. Maintaining a private Qt in my source control would not be appropriate for the DomTerm project. – Per Bothner Sep 27 '16 at 00:18
1

Since there doesn't seem to be a way to go "below" the QWebChannel (without hacking Qt itself) there seemed to be three options:

  • Create a separate WebSockets connection. That would require bigger non-local changes, and possibly creating extra processes.

  • Do the byte-to-text decoding on the C++ side, maybe using QTextDecoder. That would probably be simplest. However, there are architectural reasons for preferring to do text decoding close to where we do escape sequence processing. That is certainly more traditional, and may better handle corner cases. Plus DomTerm needs to work with other front-ends than the QtDomTerm front-end.

  • Encode the bytestream to a sequence of strings, transmit the latter using QWebChannel, convert back to bytes on the JavaScript side, and then handle text-decoding in JavaScript.

I chose to implement the third option. There is some overhead in therms of converting back-end-forth. However, I designed and implemented an encoding that is quite efficient:

/** Encode an arbitrary sequence of bytes as an ASCII string.
 * This is used because QWebChannel doesn't have a way to transmit
 * data except as strings or JSON-encoded strings.
 * We restrict the encoding to ASCII (i.e. codes less then 128)
 * to avoid excess bytes if the result is UTF-8-encoded.
 *
 * The encoding optimizes UTF-8 data, with the following byte values:
 * 0-3: 1st byte of a 2-byte sequence encoding an arbitrary 8-bit byte.
 * 4-7: 1st byte of a 2-byte sequence encoding a 2-byte UTF8 Latin-1 character.
 * 8-13: mean the same ASCII control character
 * 14: special case for ESC
 * 15: followed by 2 more bytes  encodes a 2-byte UTF8 sequence.
 * bytes 16-31: 1st byte of a 3-byte sequence encoding a 3-byte UTF8 sequence.
 * 32-127: mean the same ASCII printable character
 * The only times we generate extra bytes for a valid UTF8 sequence
 * if for code-points 0-7, 14-26, 28-31, 0x100-0x7ff.
 * A byte that is not part of a valid UTF9 sequence may need 2 bytes.
 * (A character whose encoding is partial, may also need extra bytes.)
 */
static QString encodeAsAscii(const char * buf, int len)
{
    QString str;
    const unsigned char *ptr = (const unsigned char *) buf;
    const unsigned char *end = ptr + len;
    while (ptr < end) {
        unsigned char ch = *ptr++;
        if (ch >= 32 || (ch >= 8 && ch <= 13)) {
            // Characters in the printable ascii range plus "standard C"
            // control characters are encoded as-is
            str.append(QChar(ch));
        } else if (ch == 27) {
            // Special case for ESC, encoded as '\016'
            str.append(QChar(14));
        } else if ((ch & 0xD0) == 0xC0 && end - ptr >= 1
                 && (ptr[0] & 0xC0) == 0x80) {
            // Optimization of 2-byte UTF-8 sequence
            if ((ch & 0x1C) == 0) {
                // If Latin-1 encode 110000aa,10bbbbbb as 1aa,0BBBBBBB
                // where BBBBBBB=48+bbbbbb
              str.append(4 + QChar(ch & 3));
            } else {
                // Else encode 110aaaaa,10bbbbbb as '\017',00AAAAA,0BBBBBBB
                // where AAAAAA=48+aaaaa;BBBBBBB=48+bbbbbb
                str.append(QChar(15));
                str.append(QChar(48 + (ch & 0x3F)));
            }
            str.append(QChar(48 + (*ptr++ & 0x3F)));
        } else if ((ch & 0xF0) == 0xE0 && end - ptr >= 2
                 && (ptr[0] & 0xC0) == 0x80 && (ptr[1] & 0xC0) == 0x80) {
            // Optimization of 3-byte UTF-8 sequence
            // encode 1110aaaa,10bbbbbb,10cccccc as AAAA,0BBBBBBB,0CCCCCCC
            // where AAAA=16+aaaa;BBBBBBB=48+bbbbbb;CCCCCCC=48+cccccc
            str.append(QChar(16 + (ch & 0xF)));
            str.append(QChar(48 + (*ptr++ & 0x3F)));
            str.append(QChar(48 + (*ptr++ & 0x3F)));
        } else {
            // The fall-back case - use 2 bytes for 1:
            // encode aabbbbbb as 000000aa,0BBBBBBB, where BBBBBBB=48+bbbbbb
            str.append(QChar((ch >> 6) & 3));
            str.append(QChar(48 + (ch & 0x3F)));
        }
    }
    return str;
}

Yes, this is overkill and almost certainly premature optimization, but I can't help myself :-)

Per Bothner
  • 51
  • 4