How to correctly set QWebEngine HTTP headers in a RequestInterceptor

Question

I've been encountering a problem with PyQt5's QWebEngineUrlRequestInterceptor on Python3 and, more importantly, the setHttpHeader function. Here's my code:

class WebEngineUrlRequestInterceptor(QWebEngineUrlRequestInterceptor):
def __init__(self, parent=None):
    super().__init__(parent)

def interceptRequest(self, info):
    info.setHttpHeader("X-Frame-Options", "ALLOWALL")
    print(info.requestUrl())

Unfortunately, the proper way to use this function seems to be located absolutely nowhere and as such I have had to resort to trying every possible method I can think of, to no avail.

I have also tried surrounding the arguments of setHttpHeader with QByteArray, which caused QByteArray to give me this complaint...

    Traceback (most recent call last):
  File "test.py", line 30, in interceptRequest
    info.setHttpHeader(QByteArray("X-Frame-Options"), QByteArray("ALLOWALL"))
TypeError: arguments did not match any overloaded call:
  QByteArray(): too many arguments
  QByteArray(int, str): argument 1 has unexpected type 'str'
  QByteArray(Union[QByteArray, bytes, bytearray]): argument 1 has unexpected type 'str'

I have also tried encoding the strings with .encode('ascii') and even .encode('utf-8'). While neither raised an error, the header also refused to change, which leads me to believe that the returned values are not compatible with the function.

UPDATE: Even QByteArray(b"X-Frame-Options") does not set the header. js: Refused to display 'https://www.google.co.uk/?gfe_rd=cr&dcr=0&ei=rX2gWtDJL8aN8Qfv3am4Bw' in a frame because it set 'X-Frame-Options' to 'SAMEORIGIN'. is the error I get from WebEngine.

A note to add, I am 100% sure that interceptRequest is being called. I can see the output of the print call in the terminal.

Full MCVE code at [UPDATED LINK]: https://paste.ee/p/Y0mRs

Answer 1

So first of all, the question is why is the existing code not working?

class WebEngineUrlRequestInterceptor(QWebEngineUrlRequestInterceptor):
    def __init__(self, parent=None):
        super().__init__(parent)

    def interceptRequest(self, info):
        info.setHttpHeader("X-Frame-Options", "ALLOWALL")
        print(info.requestUrl())

Now when you install a UrlRequestInterceptor, it is by all means a request interceptor. A request initiated by the WebEngineView is passed through this, you can do a lot with it

• Change the URL all together
• Block it from downloading (AdBlocking etc...)
• Add more headers to request
• Redirect to a different url

Now when you have info.setHttpHeader("X-Frame-Options", "ALLOWALL"), it is adding it to the request and not to the response. This can be verified by changing the url to http://postman-echo.com/get and you will get the below response

{
  "args": {
    
  },
  "headers": {
    "host": "postman-echo.com",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    "accept-encoding": "gzip, deflate",
    "cookie": "sails.sid=s%3AXNukTzCE5ucYNEv_NB8ULCf4esVES3cW.%2BmpA77H2%2F%2B6YcnypvZ7I8RQFvVJrdOFs8GD%2FPymF0Eo",
    "if-none-match": "W/\"1e1-rYSDjZun8qsI1ZojoxMuVg\"",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.10.1 Chrome/61.0.3163.140 Safari/537.36",
    "x-frame-options": "ALLOW",
    "x-forwarded-port": "80",
    "x-forwarded-proto": "http"
  },
  "url": "http://postman-echo.com/get"
}

But nothing is changed on the response side, you still have whatever was actually returned by the original request.

With QWebView it was possible to install a QNetworkAccessManager and return a QNetworkReply with a modified response. Something shown in

How to write content to QNetworkReply (was: QWebview with generated content)

But if you read the Porting from Qt WebKit to Qt WebEngine guide, there is a important difference to note

Qt WebEngine Does Not Interact with QNetworkAccessManager

Some classes of Qt Network such as QAuthenticator were reused for their interface but, unlike Qt WebKit, Qt WebEngine has its own HTTP implementation and cannot go through a QNetworkAccessManager.

The signals and methods of QNetworkAccessManager that are still supported were moved to the QWebEnginePage class.

I dug lot of thread asking for a response modification approach. All un-answered unfortunately

Capture server response with QWebEngineView

QWebEngineView modify web content before render

https://forum.qt.io/topic/81450/capture-client-request-headers-with-qwebengineview

Intercept AJAX POST request and read data using QWebEngine?

So it is not easily possible. But there is one workaround that I think would work, but I am not able to yet validate it

The approach is to add a new scheme url handler

self.conn_handler = AppSchemeHandler()
self.profile.installUrlSchemeHandler("conapp".encode(), self.conn_handler)
self.webpage = MyQWebEnginePage(self.profile, self.view)

Now we update the interceptor so it modifies the google url to redirect the request to our handler

class WebEngineUrlRequestInterceptor(QWebEngineUrlRequestInterceptor):
    def __init__(self, parent=None):
        super().__init__(parent)

    def interceptRequest(self, info):
        info.setHttpHeader(b'x-frame-options', b'ALLOW')
        print(info.requestUrl())

        if str(info.requestUrl().host()) == "google.com":
            url = info.requestUrl().toString()
            item = url.split("/")[-1]

            info.redirect(QUrl(r"conapp://webresource?url=" + url))

And then in our scheme handler

class AppSchemeHandler(QWebEngineUrlSchemeHandler):
    def __init__(self, parent=None):
        super().__init__(parent)

    def requestStarted(self, request):
        url = request.requestUrl().toString().replace("conapp://webresource?url=", "")
        response = QWebEngineHttpRequest(QUrl(url))

        # Do something here which returns the response back to the url

The part where we read the response and send it back is something I have not found an example of yet anywhere