I need to parse a lot of pages and at some point I get an error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /catalog/product-category/channels/videos/121 (Caused by SSLError(SSLEOFError(8
, 'EOF occurred in violation of protocol (_ssl.c:997)')))
I tried adding the following, but still the same error:
import urllib3
requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL:@SECLEVEL=1'
my code example:
for page in range(1, pages):
response = requests.get(category + f"/{page}", headers={"User-Agent": ua.random}, verify=False)
soup = BeautifulSoup(response.text, "lxml")
...
log error:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/parser/main.py", line 44, in start
self.parse_urls()
File "/parser/main.py", line 35, in parse_urls
response = requests.get(category + f"/{page}", headers={"User-Agent": ua.random})
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /catalog/product-category/channels/videos/116 (Caused by SSLError(SSLEOFError(8
, 'EOF occurred in violation of protocol (_ssl.c:997)')))