0

I would like to ask you some favor.I am trying to solve ssl error from a python login source code and still don't know how to solve this issue.

■environment

Python 3.5.2

requests -the latest version

■source code

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

# login username and password
USER = "JS-TESTER"
PASS = "ipCU12ySxI"

# session starts
session = requests.session()

# login information
login_info = {
    "username_mmlbbs6": USER,  
    "password_mmlbbs6": PASS,  
    "back": "index.php",       
    "mml_id": "0"             
}
url_login = "http://uta.pw/sakusibbs/users.php?action=login&m=try"
res = session.post(url_login, data=login_info)
res.raise_for_status() 

# pick up mypage's url
soup = BeautifulSoup(res.text, "html.parser")
a = soup.select_one(".islogin a")
if a is None:
    print("Couldn't load mypage data")
    quit()
# Convert the relative URL to an absolute URL
url_mypage = urljoin(url_login, a.attrs["href"])
print("マイページ=", url_mypage)

# Access mypage
res = session.get(url_mypage)
res.raise_for_status()

# List up favorite titles
soup = BeautifulSoup(res.text, "html.parser")
links = soup.select("#favlist li > a")
for a in links:
    href = a.attrs["href"]
    title = a.get_text()
    print("-", title, ">", href)

■error details

[vagrant@localhost ch2]$ python login-getfav.py
Traceback (most recent call last):
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
cnx.do_handshake()
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1806, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1546, in _raise_ssl_error
_raise_current_error()
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/connectionpool.py", line 850, in validate_conn
conn.connect()
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/util/ssl.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 448, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='uta.pw', port=443): Max retries exceeded with url: /sakusibbs/users.php?action=login&m=try (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "login-getfav.py", line 21, in 
res = session.post(url_login, data=login_info)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 555, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 640, in send
history = [resp for resp in gen] if allow_redirects else []
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 640, in 
history = [resp for resp in gen] if allow_redirects else []
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 218, in resolve_redirects
**adapter_kwargs
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='uta.pw', port=443): Max retries exceeded with url: /sakusibbs/users.php?action=login&m=try (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

■hypothesis

In other source code I have tried to change requests get method as follows(verify=False) and it worked. I wonder if I can apply this to session method too? is so, where should I put verify=False...

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

# login username and password
USER = "JS-TESTER"
PASS = "ipCU12ySxI"

# session starts
session = requests.session()

# login information
login_info = {
    "username_mmlbbs6": USER,  
    "password_mmlbbs6": PASS,  
    "back": "index.php",       
    "mml_id": "0"             
}
url_login = "http://uta.pw/sakusibbs/users.php?action=login&m=try"
res = session.post(url_login, data=login_info, verify=False)
res.raise_for_status() 

# pick up mypage's url
soup = BeautifulSoup(res.text, "html.parser")
a = soup.select_one(".islogin a")
if a is None:
    print("Couldn't load mypage data")
    quit()
# Convert the relative URL to an absolute URL
url_mypage = urljoin(url_login, a.attrs["href"])
print("マイページ=", url_mypage)

# Access mypage
res = session.get(url_mypage)
res.raise_for_status()

# List up favorite titles
soup = BeautifulSoup(res.text, "html.parser")
links = soup.select("#favlist li > a")
for a in links:
    href = a.attrs["href"]
    title = a.get_text()
    print("-", title, ">", href)

■error details

[vagrant@localhost ch2]$ python login-getfav.py
/home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Couldn't load mypage data
rikko215
  • 1
  • 2
  • `verify=False` means do disable any validation on the certificate. This makes the connection open to undetected man in the middle attacks. If you do this you should ask yourself why you use HTTPS in the first place because you essentially abandon the protections offered by HTTPS. – Steffen Ullrich Feb 11 '18 at 05:58
  • Apart from that, the site has a correct certificate and chain. Given that the certificate validation fails for you there is probably some problem with the trust store you are using. Try to install the `certifi` python package to get a trust store which includes the needed CA certificates. – Steffen Ullrich Feb 11 '18 at 06:10
  • Thank you so much for a quick response. I got it, using {verify=False} isn't the great idea. However it seems I've already installed certifi python package to my local. Is there a way to investigate if I am still using the wrong trust store? [vagrant@localhost ch2]$ pip install certifi Requirement already satisfied: certifi in /home/vagrant/.pyenv/versions/3.5.2/lib/python3.5/site-packages – rikko215 Feb 11 '18 at 07:28
  • In this case I suspect that some SSL interception is taking place in your environment. If you have openssl installed run `openssl s_client -connect uta.pw -servername uta.pw` and provide the output (especially the information about the issuer or the whole certificate). If you don't have the openssl binary installed try a python script as described [here](https://stackoverflow.com/questions/45478536/python-getting-common-name-from-url-using-ssl-getpeercert) to get to the certificate details. – Steffen Ullrich Feb 11 '18 at 07:43
  • I did try a python script as described in URL= https://stackoverflow.com/questions/45478536/python-getting-common-name-from-url-using-ssl-getpeercert and got result as{CN=*.cds.ca}. Umm... – rikko215 Feb 11 '18 at 11:11
  • Please take a closer look at the script you've used and what it does. You need to adapt the hostname in the script to the one you use, i.e. from `cds.ca` to `uta.pw` – Steffen Ullrich Feb 11 '18 at 11:30
  • thanks for a quick reply. I have changed it and it got a result as follows {CN=uta.pw} – rikko215 Feb 11 '18 at 11:36
  • I see, the script currently returns only the CN which fits. Could you please return the full certificate with a modified version of the script you'll find in https://pastebin.com/wLGuaEis and provide the output (in your question or maybe as pastebin too). – Steffen Ullrich Feb 11 '18 at 11:48
  • I run the script got this error..... Traceback (most recent call last): File "openSsl.py", line 7, in s.connect(dst) socket.gaierror: [Errno -2] Name or service not known – rikko215 Feb 11 '18 at 11:55
  • This is strange since the only change to the script you've used before is that it outputs the certificate instead of only the CN (i.e. last line of script). Please try again and make sure that the target server is actually reachable. Also make sure that you run all these tests in exactly the same environment in that you have the reported certificate validation problems. – Steffen Ullrich Feb 11 '18 at 11:58
  • Yeah it is strange but it did'nt work. so I have added the last line `print(crypto.dump_certificate(crypto.FILETYPE_PEM, x509))` to [link](https://pastebin.com/wLGuaEis) and then it worked. the result i got from the script was as follows – rikko215 Feb 11 '18 at 12:15
  • b'-----BEGIN CERTIFICATE-----\nMIIE+zCCA+OgAwIBAgISA6IsrUuolYmbyA0a5GVwP2BxMA0GCSqGSIb3DQEBCwUA\nMEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD\nExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODAyMDUxNTMwMTZaFw0x\nODA1MDYxNTMwMTZaMBExDzANBgNVBAMTBnV0YS5wdzCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAJlab00jEk2Pzy/24VLtSA2+OPgf0MTqtJAX+B7ZNB8H\nJGRLg0US4OtQcCYKVdBPE+D0B39XXq0Y9ZoF4WMtLE3jBv2v4vaeg8nvAG5An/Lv\niNInvTYzG+8qpEwbR9zSts2nUGkhhW5x/ – rikko215 Feb 11 '18 at 12:21
  • sOE6OTPLCD22BzTf8VTXyQyzWo0muGp\nSUo5ijplqKTsKOniHWCI/mkbZJBfwgXD0CvfB3l7sBuRGZz8zSit+KZvN6aU1096\nZ1WIJOn2Q7SZs/Q4ohz4y8qLmaW1IOwSPZfxmhfOgAOr6wiFlQS1cnnz28DDoeA3\na7wn1bhZ3rwwTt3tbFAEVw/kbif60oX9QOItcJf4SesCAwEAAaOCAhIwggIOMA4G\nA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD\nVR0TAQH/BAIwADAdBgNVHQ4EFgQU4XDWlmSZ1cS+kQHbYzbqic0oRlwwHwYDVR0j\nBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUHAQEEYzBhMC4GCCsG\nAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5cHQub3JnMC8GCCsG\nAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5cHQub3JnLzAdBgNV\nHREEFjAUggZ – rikko215 Feb 11 '18 at 12:21
  • 1dGEucHeCCnd3dy51dGEucHcwgf4GA1UdIASB9jCB8zAIBgZngQwB\nAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxl\ndHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRpZmljYXRl\nIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGllcyBhbmQg\nb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBvbGljeSBm\nb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5LzANBgkq\nhkiG9w0BAQsFAAOCAQEAMeV3SaNYfMXq9f/RQ/ZhIGqNlrymBAFkF6LreYVI4EQy\nAhhIUV2Dj+S – rikko215 Feb 11 '18 at 12:21
  • xy5njwNIqqf1HsWmjQ5a5hYsPGwgOIm2OCCg2oj6ORSVwbHIaLLUK\nZ4isi1M2oC1K7kQ4jl17ll1AkInp0yi8zPYSQEgG35NTB8KrmpwIeha2+zRUfmu5\ni3FfqjYZ315gh0niu9KOUsRx1nfX4gr5NVpVofyJRJHIDk133V7bWNmWCdR3XmP5\n75iQpsEtA9Hb7lH1EXGoIzT6Js8YvPr99EgeOlDwkzU7uaN3kBfWtGNYiN4zaHwe\n4KXjfolO6C2GX8f7QkaMc6cUPzAt2OJVxUvEpconRg==\n-----END CERTIFICATE-----\n' – rikko215 Feb 11 '18 at 12:22
  • Strange: this is exactly the certificate as expected. If you really got this output from the same environment as you have the problem and you have a current certifi it should work as long as requests is actually using certifi - which is does since requests version 2.4.0 (released 4 years ago). – Steffen Ullrich Feb 11 '18 at 12:29
  • okay... thanks...since my environment is on CentOS, I might just switch it to Ubuntu and see what happens. – rikko215 Feb 11 '18 at 12:38

0 Answers0