2

This question just recently posted has some useful answers, but it's not the same as mine. I'm running urllib3 1.26.4 and Python 3.7 from an ArcGIS Pro Notebook. I also have Fiddler 4 open because I want to track web traffic while troubleshooting a script. I only get the following error when I have Fiddler open. If I close Fiddler I get <Response [200]>. Is it not possible to use the requests module with Fiddler open? I'm new to Fiddler.

Truncated script:

import requests

#url
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'
#headers
headers = {'user-agent': 'Mozilla/5.0'}

response = requests.get(idph_data, headers=headers, verify=True)

Error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
In  [35]:
Line 4:     response = requests.get(idph_data,verify=True)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py, in get:
Line 76:    return request('get', url, params=params, **kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py, in request:
Line 61:    return session.request(method=method, url=url, **kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py, in request:
Line 542:   resp = self.send(prep, **send_kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py, in send:
Line 655:   r = adapter.send(request, **kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\adapters.py, in send:
Line 449:   timeout=timeout

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connectionpool.py, in urlopen:
Line 696:   self._prepare_proxy(conn)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connectionpool.py, in _prepare_proxy:
Line 964:   conn.connect()

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connection.py, in connect:
Line 359:   conn = self._connect_tls_proxy(hostname, conn)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connection.py, in _connect_tls_proxy:
Line 506:   ssl_context=ssl_context,

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\util\ssl_.py, in ssl_wrap_socket:
Line 432:   ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\util\ssl_.py, in _ssl_wrap_socket_impl:
Line 474:   return ssl_context.wrap_socket(sock)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py, in wrap_socket:
Line 423:   session=session

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py, in _create:
Line 827:   raise ValueError("check_hostname requires server_hostname")

ValueError: check_hostname requires server_hostname
---------------------------------------------------------------------------
Pfalbaum
  • 586
  • 3
  • 10
  • 26
  • 1
    Arcgispro seems to implement an custom SSL/TLS cert validation in the file `C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py`. Therefore it does not trust the custom root CA certificate used by Fiddler. – Robert Apr 01 '21 at 14:14
  • Did you find a solution to this ? – dparkar Apr 14 '21 at 21:16
  • Not directly, no. However, I've been experimenting with Wireshark:https://www.wireshark.org/, which does not cause that error. – Pfalbaum Apr 15 '21 at 18:51
  • I installed a FiddlerRoot certificate, but I still get the error. In Fiddler: Tools-->Options...-->HTTPS-->Actions-->Export Root Certificate to Desktop-->Install Certificate. – Pfalbaum Apr 28 '21 at 14:22

2 Answers2

2

I am running into this issue as well with the environment provided by the current version of ArcGIS Pro. Per a lower-rated answer in the question you linked, I ran pip install urllib3==1.25.11 in the desired environment (in my case a clone of the default), and the issue appears to be resolved.

This is apparently due to a new feature in the urllib3 version provided by ArcGIS Pro. The above command downgrades to a relatively recent, but working, version. This will not be resolved in newer versions of urllib3, but instead, there is currently a pull request pending to fix the underlying issue in Python itself.

By the way, while it's possible to configure pip to be able to run through the fiddler proxy, it's not too easy, so it is best to turn off Fiddler while running any pip commands.

The pertinent bug report is found here. The issue appears to be that there is a very old bug in how Windows system proxy settings are being parsed by CPython / built-in urllib, causing the proxy entry for use with https URLs to always receive a HTTPS prefix (instead of HTTP). Newer version of urllib3 actually support using proxies over HTTPS, which was not previously the case. So before, urllib3 would ignore the prefix, but now, it attempts to use HTTPS to communicate with a HTTP url.

PhilippNagel
  • 70
  • 1
  • 15
  • Do you have a link to this bug? – Pfalbaum May 04 '21 at 19:39
  • 1
    You gave me the idea to run my script in an Anaconda environment/Python 3.8. I did this while running ```urllib v. 1.25.11```, and with Fiddler running. This time it captured the traffic it appears. So, this also gives your observation some credit. I'm waiting to see if anybody else has any insight. – Pfalbaum May 04 '21 at 21:20
  • 1
    I saw this called a bug somewhere else, but there was also no reference for that. So I might be mistakenly calling it a bug, when it really is just a change in urllub3's API that has not been addressed by some dependency. – PhilippNagel May 05 '21 at 01:38
  • 1
    @Pfalbaum I believe this is the pertinent bug: https://bugs.python.org/issue42627 and it looks like there is a pending pull request (https://github.com/python/cpython/pull/26307), but unfortunately, the fix seems to be in CPython, so not necessarily as easy a fix as upgrading urllib3 or requests. But the urllib3 downgrade should continue to work. – PhilippNagel Jul 01 '21 at 18:47
1

I've updated to requests v. 2.7.0, the latest, and I'm no longer receiving the error. If it was a version-specific issue related to v. 2.25.1, which was what I was using, I'm not sure. I haven't came across any evidence of that.

In a Windows command prompt in the same directory as my Python executable:

python -m pip install requests==2.7.0

Now if I run my original script with Fiddler capturing, I get a HTTP status of 200 and my script no longer gives me the error.

Pfalbaum
  • 586
  • 3
  • 10
  • 26