0

I need to parse a lot of pages and at some point I get an error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /catalog/product-category/channels/videos/121 (Caused by SSLError(SSLEOFError(8
, 'EOF occurred in violation of protocol (_ssl.c:997)')))

I tried adding the following, but still the same error:

import urllib3

requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL:@SECLEVEL=1'

my code example:

for page in range(1, pages):
    response = requests.get(category + f"/{page}", headers={"User-Agent": ua.random}, verify=False)
    soup = BeautifulSoup(response.text, "lxml")
    ...

log error:

During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
     self.run()
   File "/usr/local/lib/python3.10/threading.py", line 953, in run
     self._target(*self._args, **self._kwargs)
   File "/parser/main.py", line 44, in start
     self.parse_urls()
   File "/parser/main.py", line 35, in parse_urls
     response = requests.get(category + f"/{page}", headers={"User-Agent": ua.random})
   File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get
     return request("get", url, params=params, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
     r = adapter.send(request, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /catalog/product-category/channels/videos/116 (Caused by SSLError(SSLEOFError(8
, 'EOF occurred in violation of protocol (_ssl.c:997)')))
  • Probably this: https://stackoverflow.com/questions/33410577/python-requests-exceptions-sslerror-eof-occurred-in-violation-of-protocol –  Dec 28 '22 at 14:48

0 Answers0