Use urllib or requests to get data from URL

Question

I am trying to use urllib, urllib2 or requests in order to get specific data from a site. I use Python2.7. I keep getting an error:

requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)

This is my code with requests:

  import requests
  r = requests.get("https://metoc.ndbc.noaa.gov/jtwc#_48_INSTANCE_0SiamlX2KcM6_=https%3A%2F%2Fmetoc.ndbc.noaa.gov%2FProductFeeds-portlet%2Fimg%2Fjtwc%2Fhtml%2Fcoop.jsp%3F")

I have tried everything but nothing works:

I tried to set the verification parameter to false. I get the same error. I tried to use an adaptor (as described in other answers) in order to use another SSL version. I still get the same error.

Any ideas what can the issue be? Can it be that the host has locked the access and there is no possibility to get the source for this URL?

What sub-version of Python 2.7 are you using, and on what version of what OS? — jwodder, Jul 06 '16 at 17:25
Guys this is really strange. I use python 2.7.10. We have tried these in 3 different computers. All of them in macs with Yosemite or ElCapitan. Can this be related? — user1919, Jul 06 '16 at 17:27
Version of Elcapitan is 10.11.5. Here you can see similar error: https://github.com/kennethreitz/requests/issues/2627 — user1919, Jul 06 '16 at 17:31
running `openssl s_client -connect metoc.ndbc.noaa.gov:443 -ssl3` `SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:339` — Padraic Cunningham, Jul 06 '16 at 18:41
also mounting a custom adapter http://docs.python-requests.org/en/master/user/advanced/#example-specific-ssl-version and issuing the request gives me the same error in python, try explicitly setting the version to tls. http://pastebin.com/WUsRwqMj FWIW `ssl_version=ssl.PROTOCOL_SSLv23` also works — Padraic Cunningham, Jul 06 '16 at 18:46
@user1919 You get the same sslv3 error explicitly setting tls? The server will not accept any sslv3 even with ECDHE encryption — Padraic Cunningham, Jul 06 '16 at 23:02
@Padraic Cunningham yup. I get the same error although I set tls. I can post my code tomorrow when I go at work. — user1919, Jul 06 '16 at 23:05
But more or less I followed this here: https://lukasa.co.uk/2013/01/Choosing_SSL_Version_In_Requests/ — user1919, Jul 06 '16 at 23:06
Maybe a silly question but you did use s to make the request ? i.e the session object? — Padraic Cunningham, Jul 06 '16 at 23:07
@Padraic Cunningham tomorrow I will try to do what described here: http://techiezone.rottigni.net/2016/03/python-2-7-11-on-el-capitan-with-tls-1-2-support/ — user1919, Jul 06 '16 at 23:09
One last thing what does `openssl version` output from bash? — Padraic Cunningham, Jul 06 '16 at 23:12
OK then we have the problem, you are still using the old openssl version, from memory I downloaded the binary and just replaced the /usr/bin/openssl with the new. I imagine brew may install in somewhere like usr/local or the mac equivalent — Padraic Cunningham, Jul 06 '16 at 23:15
So it would be ok if I remove the existing openssl from /usr/bin and copy there the one I installed with brew? Or download the binary one from ftp://ftp.openssl.org/source/ Right? — user1919, Jul 06 '16 at 23:20
I downloaded the binary from the ftp site and just did a sudo mv etc.., there shouldn't be any logocla reason that it would cause any issue, that is exactly what I did when i wne tfrom 0.9.8 to 1.0.1f — Padraic Cunningham, Jul 06 '16 at 23:28

score 2 · Accepted Answer · answered Jul 06 '16 at 20:57

2

All of them in macs with Yosemite or ElCapitan...

These versions still come with OpenSSL 0.9.8 so my guess is that your python is linked against this version of OpenSSL (python does not use the native TLS stack in OS X). But since the server only supports ECDHE ciphers and these are not supported by OpenSSL 0.9.8 the handshake will fail. See the report from SSLLabs for more details.

answered Jul 06 '16 at 20:57

Steffen Ullrich

114,247
10
131
172

Thanks! So what would be my options here? – user1919 Jul 06 '16 at 22:47
@user1919 upgrade your openssl ftp://ftp.openssl.org/source/ – Padraic Cunningham Jul 06 '16 at 22:57
@Padraic Cunningham I have updated my openssl using brew. This didn't solve the issue! – user1919 Jul 06 '16 at 22:59
1

Did you try setting the adapter using tls as per the link I added to the comments? – Padraic Cunningham Jul 06 '16 at 23:00
1

@user1919: it is not enough to update openssl but it has to be linked with python. To check what version of openssl your python is using check `ssl.OPENSSL_VERSION`. See http://stackoverflow.com/questions/18752409/updating-openssl-in-python-2-7 for more information for how to update your python to include a newer version of openssl. – Steffen Ullrich Jul 07 '16 at 05:16
@Steffen Ullrich thank you. The procedure described in the link made things work! – user1919 Jul 07 '16 at 08:08

Use urllib or requests to get data from URL

1 Answers1