isinstance() unexpectedly returning False

Question

I am using the kubernetes python client. In the event that kubernetes isn't available when my code starts up, I would like to retry the connection.

When the client is unable to connect, it throws what appears to be a urllib3.exceptions.MaxRetryError exception, so I started with something like this:

import time
import urllib3

import kubernetes

kubernetes.config.load_kube_config()
api = kubernetes.client.CoreV1Api()

while True:
    try:
        w = kubernetes.watch.Watch()
        for event in w.stream(api.list_pod_for_all_namespaces):
            print event
    except urllib3.exceptions.HTTPError:
        print('retrying in 1 second')
        time.sleep(1)

But that completely fails; it acts like there is no except statement and bails out with:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2743110>: Failed to establish a new connection: [Errno 111] Connection refused',))

I thought maybe I didn't understand inheritance as well as I thought, so I replace the above with:

except urllib3.exceptions.MaxRetryError:
    print('retrying in 1 second')
    time.sleep(1)

Which fails in the same way. In an effort to figure out what was going on, I added a catch-all except and invoked pdb:

except Exception as err:
    import pdb; pdb.set_trace()

And from the pdb prompt, we can see:

(Pdb) type(err)
<class 'urllib3.exceptions.MaxRetryError'>

...which looks fine, as does the mro:

(Pdb) import inspect
(Pdb) inspect.getmro(err.__class__)
(<class 'urllib3.exceptions.MaxRetryError'>, <class 'urllib3.exceptions.RequestError'>, <class 'urllib3.exceptions.PoolError'>, <class 'urllib3.exceptions.HTTPError'>, <type 'exceptions.Exception'>, <type 'exceptions.BaseException'>, <type 'object'>)

But despite all that:

(Pdb) isinstance(err, urllib3.exceptions.MaxRetryError)
False

And all the paths look reasonable:

(Pdb) urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'
(Pdb) kubernetes.client.rest.urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'

So...what the actual what is going on here?

Update

Here is the full stack trace:

Traceback (most recent call last):
  File "testkube.py", line 13, in <module>
    for event in w.stream(api.list_pod_for_all_namespaces):
  File "/usr/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 116, in stream
    resp = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14368, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14464, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
    _preload_content, _request_timeout)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
    headers=headers)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
    query_params=query_params)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 223, in request
    headers=headers)
  File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x3d16110>: Failed to establish a new connection: [Errno 111] Connection refused',))

What happens if you import `urllib3` _after_ importing `kubernetes`? — Mark Dickinson, Oct 12 '17 at 17:40
@MarkDickinson changing the order of the imports doesn't appear to change the behavior. — larsks, Oct 12 '17 at 17:41
@larsks Thanks. It was a long shot. I was wondering whether either kubernetes or one of its dependencies (e.g., requests) was monkeypatching urllib3. — Mark Dickinson, Oct 12 '17 at 17:43
Hm. It's not happening in a different thread or anything like that. It looks like you have multiple copies of the same class, almost as if a module got reloaded. You didn't reload any modules, did you? — user2357112, Oct 12 '17 at 17:45
@user2357112 I am simply running the code presented in this question (the code is in `testkube.py` and I am running `python testkube.py`). — larsks, Oct 12 '17 at 17:47
There does seem to be something funky with my environment: if I create a clean virtualenv I cannot reproduce this behavior. I don't mind "my environment is screwy" as an answer, but I would love to find a positive diagnostic for this issue. — larsks, Oct 12 '17 at 17:49
Try running `python -v testkube.py` to turn on [import logging](https://docs.python.org/3/using/cmdline.html#id4) and see if it looks like any modules are getting initialized twice. — user2357112, Oct 12 '17 at 17:51
@user2357112 that produces some interesting but lengthy results. I've put them [here](http://chunk.io/larsks/a514bf26a4af46e7985610b2cc0caa65) for now, although I would be happy to put an excerpt in the question. I see that there is a `requests.packages.urllib3` in the mix, which I assume is the source of the problem, although it's not clear to me exactly where this is happening. — larsks, Oct 12 '17 at 17:57
@larsks: Looks like chunk.io is blocked here, supposedly for "potential malicious activity or other security reasons". Maybe Pastebin? — user2357112, Oct 12 '17 at 18:01
The line `import requests.packages.urllib3.exceptions # precompiled from /usr/lib/python2.7/site-packages/urllib3/exceptions.pyc` is _very_ suspicious ... — Mark Dickinson, Oct 12 '17 at 18:03
That's coming from [here](https://github.com/requests/requests/blob/master/requests/packages.py). It's definitely suspicious, but I'm not yet seeing a way that it could be causing this issue - it doesn't seem to generate any duplicate module objects or reexecute class definitions, for example. — user2357112, Oct 12 '17 at 18:05
@user2357112 [here](https://paste.fedoraproject.org/paste/Vw4vVeR0DBsbgyOulksV~w) is another link for the `python -v` output. — larsks, Oct 12 '17 at 18:07
Hmm... that inner loop over `list(sys.modules)` in `request.packages` won't find submodules that haven't been loaded yet. That could cause problems. Might be dependent on package versions... — user2357112, Oct 12 '17 at 18:12
Older versions of `requests` initialize `requests.packages` differently, and on those versions, `requests.packages` *does* cause reloading of `urllib3.exceptions`. `requests.packages` *was* the culprit! — user2357112, Oct 12 '17 at 18:32

user2357112 · Accepted Answer · 2017-10-14T18:29:41.697

Your code has an indirect dependency on the requests package, and the requests package has a strange submodule called requests.packages. This used to contain copied source code from a number of dependencies, including urllib3, but they stopped doing that. They wanted to keep requests.packages around for backward compatibility, though, so now they're doing something weird.

Instead of requests.packages including a complete copy of the urllib3 source code, it now imports urllib3 and sets sys.modules['requests.packages.urllib3'] = urllib3. Depending on the requests version, it may set a number of other sys.modules entries, too; for example, as of requests 2.18.4, the source code does

for package in ('urllib3', 'idna', 'chardet'):
    locals()[package] = __import__(package)
    # This traversal is apparently necessary such that the identities are
    # preserved (requests.packages.urllib3.* is urllib3.*)
    for mod in list(sys.modules):
        if mod == package or mod.startswith(package + '.'):
            sys.modules['requests.packages.' + mod] = sys.modules[mod]

but in 2.17.0, it does

import urllib3
sys.modules['requests.packages.urllib3'] = urllib3

import idna
sys.modules['requests.packages.idna'] = idna

import chardet
sys.modules['requests.packages.chardet'] = chardet

This code interacts badly with submodules of the imported packages. If some code tries to do import requests.packages.urllib3.exceptions and Python doesn't find a sys.modules['requests.packages.urllib3.exceptions'] entry, Python will recreate the urllib3.exceptions module and set urllib3.exceptions and sys.modules['requests.packages.urllib3.exceptions'] to the new module (but it won't touch sys.modules['urllib3.exceptions']. This will generate new copies of the classes involved, causing your error.

A related problem with the same cause was reported back in May, leading to the new code shown in 2.18.4. 2.18.4 shouldn't cause the specific problems you're seeing, but it's still fragile, because if any submodules of urllib3 aren't yet loaded at the time requests.packages screws with sys.modules, those submodules will exhibit the same problems you've seen today.

Thank you for your help figuring that out! I was mystified. – larsks Oct 12 '17 at 18:53 — larsks, Oct 12 '17 at 18:53

isinstance() unexpectedly returning False

1 Answers1

Linked