4

I am using Twisted (16.3) and Treq (15.1) to make async requests in Python (2.7).

I am having issues with some requests over HTTPS.

Some sites have an invalid certificate and thus when making a request to them, I get this:

twisted.python.failure.Failure OpenSSL.SSL.Error

I want my client to trust any server, including those without certificates or with self signed certificates.

How can I disable certificate checks on my client?

This is a question essentially identical to mine: https://stackoverflow.com/questions/34357439/ssl-options-for-twisted-agents

Thanks!

Community
  • 1
  • 1
mayk93
  • 1,489
  • 3
  • 17
  • 31

2 Answers2

6

Update Feb 7, 2019

Here's a simple way to make a domain whitelist for treq

from treq.client import HTTPClient
from twisted.web.iweb import IPolicyForHTTPS
from twisted.web.client import BrowserLikePolicyForHTTPS, Agent
from twisted.internet.ssl import CertificateOptions
from twisted.internet import task, defer, ssl
from zope.interface import implementer

@implementer(IPolicyForHTTPS)
class WhitelistContextFactory(object):
    def __init__(self, good_domains=None):
        """
        :param good_domains: List of domains. The URLs must be in bytes
        """
        if not good_domains:
            self.good_domains = []
        else:
            self.good_domains = good_domains

        # by default, handle requests like a browser would
        self.default_policy = BrowserLikePolicyForHTTPS()

    def creatorForNetloc(self, hostname, port):
        # check if the hostname is in the the whitelist, otherwise return the default policy
        if hostname in self.good_domains:
            return ssl.CertificateOptions(verify=False)
        return self.default_policy.creatorForNetloc(hostname, port)

@task.react
@defer.inlineCallbacks
def main(reactor):
    # make a custom client, agent, and context factory
    # NOTE: WhitelistContextFactory() takes a list of BYTES
    treq = HTTPClient(Agent(reactor, contextFactory=WhitelistContextFactory([b'example.net'])))
    response = yield treq.get('https://example.net/version')
    content = yield response.content()
    print(content)

WhitelistContextFactory takes a list of URLs (in bytes) and checks if the hostname is in the list to disregard TLS verification. You can get fancy and use regex too. Kudos to https://github.com/twisted/treq/issues/213

DO NOT DO THIS!

I've been trying to do this too over the past few days as well. With all the effort I put into circumventing certificate verification, I could've easily just created a pair of keys and been on my merry way :D. I found this comment on the treq issues board that monkeypatches the issue:

from twisted.internet import _sslverify
_sslverify.platformTrust = lambda : None

I'm sure there's a convoluted way of doing it "correctly" but it wouldn't be worth the effort in my opinion. I did a patch such that it wouldn't override platformTrust() and I'll try to get it merged but I wouldn't hold my breath. From the tone of some of the bug comments I've seen pertaining to trust roots, ssl, and certificates, I don't think it will get merged. Hope this helps though.

notorious.no
  • 4,919
  • 3
  • 20
  • 34
  • Hey! Thanks for the answer! I came across that comment as well but it did not help me. I am using agents for my requests, from `twisted.web.client import Agent`. I am guessing the code, `_sslverify.platformTrust = lambda : None` specifically overwrites platformTrust to do nothing. Do you know, is that applicable to agents? So is there something else that should get changed if agents are used? – mayk93 Sep 27 '16 at 10:43
  • This has been alluding me nearly a day now. I can't believe how bad Python / Twisted are at handling certificate validation for production/dev scenarios - is eveyone in python community using real certificates, or polluting their machines with their own CA's or just not testing SSL/TLS? – jmc Dec 01 '16 at 18:15
  • I totally agree! There are treq devs working on a "whitelist" feature which will allow select sites to go unverified. Hopefully it comes soon – notorious.no Dec 02 '16 at 04:28
  • It's good to know a whitelist feature is coming out. To be honest, at least for my project, I found it much easier to switch to python3.5 aiohttp. In my oppinion, it provides asynchronous requests in a more elegant way compared to Twisted, provides handling of SSL certificates and a much easier way to work with proxies, just to name a few things. Sure, it involves some rewriting but in the long run it's worth it. – mayk93 Dec 02 '16 at 15:21
  • Nothing wrong with being honest :) Just FYI that tornado integrates well with twisted also. You could've used tornado's ``AsyncHTTPClient.fetch()`` and then integrated the event loop w/ twisted. ``asyncio`` integration is coming soon as well. This way you wouldn't have to rewrite too much code. – notorious.no Dec 02 '16 at 19:08
1

If you want to completely ignore the check for SSL here is a workaround:

from twisted.internet import ssl, _sslverify
from twisted.web.iweb import IPolicyForHTTPS

@implementer(IPolicyForHTTPS)
class IgnoreHTTPS:
    def creatorForNetloc(self, hostname, port):
        options = ssl.CertificateOptions(verify=False)
        return _sslverify.ClientTLSOptions(hostname.decode('ascii'), options.getContext())
VladiC4T
  • 216
  • 2
  • 10