Error using AdWordsClient from pytest - Recursion detected (same locals & position)

Question

I need to write a set of tests to check if some calls to the google AdWords API are bringing the right data.

For that purpose I wrote a fixture method that yields the google client:

@pytest.fixture
def google_client():
    client = adwords.AdWordsClient.LoadFromStorage('tests/googleads.yaml')
    client.cache = common.ZeepServiceProxy.NO_CACHE

    return client

Then I have a method that tries to connect to the Google API, but even before that, just when trying to get a report downloader

def test_performance_report(google_client):
    google_client.SetClientCustomerId(*****)
    report_downloader = google_client.GetReportDownloader(version='v201809')
    >

There it breaks with this stack trace:

tests/test_googleads_api.py:12:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
env/lib/python3.6/site-packages/googleads/adwords.py:478: in GetReportDownloader
    return ReportDownloader(self, version, server)
env/lib/python3.6/site-packages/googleads/adwords.py:1321: in __init__
    self.proxy_config, self._namespace, self._adwords_client.cache)
env/lib/python3.6/site-packages/googleads/common.py:821: in __init__
    data = transport.load(endpoint)
env/lib/python3.6/site-packages/zeep/transports.py:110: in load
    content = self._load_remote_data(url)
env/lib/python3.6/site-packages/zeep/transports.py:126: in _load_remote_data
    response = self.session.get(url, timeout=self.load_timeout)
env/lib/python3.6/site-packages/requests/sessions.py:546: in get
    return self.request('GET', url, **kwargs)
env/lib/python3.6/site-packages/requests/sessions.py:533: in request
    resp = self.send(prep, **send_kwargs)
env/lib/python3.6/site-packages/requests/sessions.py:646: in send
    r = adapter.send(request, **kwargs)
env/lib/python3.6/site-packages/requests/adapters.py:449: in send
    timeout=timeout
env/lib/python3.6/site-packages/urllib3/connectionpool.py:600: in urlopen
    chunked=chunked)
env/lib/python3.6/site-packages/urllib3/connectionpool.py:343: in _make_request
    self._validate_conn(conn)
env/lib/python3.6/site-packages/urllib3/connectionpool.py:839: in _validate_conn
    conn.connect()
env/lib/python3.6/site-packages/urllib3/connection.py:332: in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
env/lib/python3.6/site-packages/urllib3/util/ssl_.py:281: in create_urllib3_context
    context.options |= options
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py:465: in options
    super(SSLContext, SSLContext).options.__set__(self, value)
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py:465: in options
    super(SSLContext, SSLContext).options.__set__(self, value)
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)

Just to note that the same code works in the production, however when running this in the test it breaks. I don't understand what's wrong, I'd appreciate any help

I'm running this with:

platform darwin -- Python 3.6.7, pytest-4.3.0, py-1.7.0, pluggy-0.8.0 plugins: flask-0.14.0, celery-4.2.0

DEBUG: If I run this from the python console it works, however from the test it doesnt:

from requests.packages.urllib3.util.ssl_ import create_urllib3_context
    create_urllib3_context()
    print(ssl.SSLContext)

Error:

env/lib/python3.6/site-packages/urllib3/util/ssl_.py:281: in create_urllib3_context
    context.options |= options
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py:465: in options
    super(SSLContext, SSLContext).options.__set__(self, value)
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py:465: in options
    super(SSLContext, SSLContext).options.__set__(self, value)
E   RecursionError: maximum recursion depth exceeded while calling a Python object
!!! Recursion detected (same locals & position)

Thanks!

UPDATE 1 I added:

import eventlet
eventlet.monkey_patch()

before anything else in my conftest.py and now I get this error:

            except OSError as err: # timeout error
>               raise URLError(err)
E               urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py:1320: URLError

UPDATE 2 The second problem is with the Python OpenSSL version, I solved it following advice from this post:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

are you using gavent? try update `pip install --upgrade requests` — M.A.K. Simanto, Mar 10 '19 at 12:36
No. I tried also upgrading requests to the latest version, no changes. — magnoz, Mar 10 '19 at 12:51
can you paste the output for # python >>> from requests.packages.urllib3.util.ssl_ import create_urllib3_context >>> create_urllib3_context() — M.A.K. Simanto, Mar 10 '19 at 13:21
Do you use `locust`? I am being agnostic about pyhton google SDK , just wondering is there any place where you need to configure url manually and you missed https ? — M.A.K. Simanto, Mar 11 '19 at 13:48
No, I'm using the google python lib directly, nothing else. It's still related to the ssl.SSLContext thing, yet don't understand what the issue is.. — magnoz, Mar 11 '19 at 15:12

score 1 · Accepted Answer · answered Mar 10 '19 at 17:40

1

This looks like a bad interaction between gevent and requests as described in this issue

Make sure if you're using gevent that you also patch it under test -- your root conftest.py is a good place to put the gevent patching code:

from gevent import monkey
monkey.patch_all()

answered Mar 10 '19 at 17:40

anthony sottile

61,815
15
148
207

Also If I run that from a clean python script like using jypiter for instance, it works. However from a pytest It doesn't. About gavent, I'm not using, however I'm not that concious about that does the Google python sdk uses inside. – magnoz Mar 11 '19 at 08:14
Add the output of `pip freeze --all` to your question. It is no surprise if didn't error from a clean script as gevent doesn't get imported – anthony sottile Mar 11 '19 at 14:49
@M.A.K. Simanto I saw the bit about "gavent" but that isn't "gevent" – anthony sottile Mar 11 '19 at 14:51
@AnthonySottile I'm actually using eventlet on my app (not in my tests). On my app I use celery, flask-socketio, eventlet and I have `eventlet.monkey_patch()` pretty much at the top of my code – magnoz Mar 11 '19 at 19:14
the `monkey_patch` call needs to be the very first thing that happens, in this case before `requests` is imported. You can probably locally reproduce your stacktrace by importing `requests`, the importing and monkeypatching, then using requests – anthony sottile Mar 11 '19 at 19:18
I added it before importing googleads (in my test) but still not working. This is that I added: `import eventlet` `eventlet.monkey_patch()` `from googleads import adwords` – magnoz Mar 12 '19 at 07:55
Add it in your root conftest, otherwise it is too late – anthony sottile Mar 12 '19 at 13:40
Right I did what you said, now I have another error, see `UPDATE 1` in the post – magnoz Mar 12 '19 at 14:46
@magnoz I would have given the same solution if you could've answered the gavent dependency at the first place when I asked. Btw good to see it got solved after investing hours on a different direction. – M.A.K. Simanto Mar 14 '19 at 12:25
@M.A.K.Simanto technically neither "gavent" nor "gevent" were being used, eventlet is basically an equivalent though – anthony sottile Mar 14 '19 at 15:52

Error using AdWordsClient from pytest - Recursion detected (same locals & position)

1 Answers1