2

I have a Perl script that uses WWW::Mechanize to connect to a site over https, and that script just stopped working the other day. The status and error message I get back are 500 and "Can't connect to jobs.illinois.edu:443". The URL I'm trying to connect to is https://jobs.illinois.edu/. I can connect from my browser (Firefox). My platform is Linux -- up-to-date Arch Linux. I can also connect (using WWW::Mechanize) to other https sites.

I tried using LWP::UserAgent, and the behavior is the same.

I'm using ssl_opts => { SSL_version => 'TLSv1' }; I don't remember why I added that -- it may have been necessary to get it working at some point.

Any ideas on how to fix this, or how I might get more information as to what the problem is? Are there other ssl options I can try?

I have a feeling there was some slight configuration change on the site that led to this problem -- maybe some SSL-protocol version change or something like that. (I don't think I updated anything on my machine inbetween the times it worked and stopped working.)

Thanks.

Here's sample code that fails:

#!/usr/bin/perl

use strict;
use warnings;

use constant AJB_URL => 'https://jobs.illinois.edu/academic-job-board';

use WWW::Mechanize;

my $mech = WWW::Mechanize->new( ssl_opts => { SSL_version => 'TLSv1' } );

$mech->get( AJB_URL );

It returns:

Error GETing https://jobs.illinois.edu/academic-job-board: Can't connect to jobs.illinois.edu:443 at ./test2.pl line 12.
mepstein
  • 133
  • 2
  • 7
  • [More info](http://pastebin.com/ZaHuCL7u), which means it can't verify the certificate for some reason. Also, the cert FF uses only came into effect a few days ago (2015-05-22T0:00:00Z), so something was changed on their end. – ikegami May 25 '15 at 06:18
  • Disabling certificate verification "works" at the cost of rendering the HTTPS useless – ikegami May 25 '15 at 06:35
  • @ikegami: I tried your code from pastebin on my system, and it doesn't fail. So apparently my issue's a little different. I did get things to work by adding $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;. Does that disable all certificate verification, or only one piece of it? – mepstein May 26 '15 at 02:57
  • It depends on the version of LWP used. It should only disable hostname verification from the name but in fact disables complete verification on your system. And even just disabling host name verification allows the same attacks as completely disabling verification, because then the man-in-the-middle could just use any certificate signed by trusted CA, i.e. also a certificate for his own domain. – Steffen Ullrich May 26 '15 at 04:20

1 Answers1

7

... that script just stopped working the other day.

Which in most cases is caused by server-side or client-side changes. But I assume that you did not make any changes on the client side.

Calling your code with perl -MIO::Socket::SSL=debug4... gives:

DEBUG: ...SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Looking at the SSLLabs report you see two trust paths, where one requires an extra download. The root-CA "USERTrust RSA Certification Authority" for the first trust path is not installed on my system (Ubuntu 14.04), and I guess it is not installed on yours (no information about your OS is known, so just guessing). This means the second trust chain will be used and the relevant Root-CA "AddTrust External CA Root" is also installed on my system. Unfortunately this trust chain is missing an intermediate certificate ("Extra download"), so the verification fails.

To fix the problem, find the missing root-CA which should match the fingerprint 2b8f1b57330dbba2d07a6c51f70ee90ddab9ad8e and use it:

$ENV{PERL_LWP_SSL_CA_FILE} = '2b8f1b57330dbba2d07a6c51f70ee90ddab9ad8e.pem';

Looking at the certificate you see that it was issued on 22 May 2015, i.e. three days ago. This explains why the problem happened just now.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Steffen Ullrich
  • 114,247
  • 10
  • 131
  • 172
  • OK, thanks. That basically makes sense, but things still aren't quite clear. For one thing, my system does have both those root CA certs (in /etc/ssl/certs). (My system is Arch Linux, up-to-date, as of a few days ago -- I did mention that in my original post.) Second, I tried adding that line specifying PERL_LWP_SSL_CA_FILE, and it didn't help (it said "SSL_ca_file ... does not exist"). Third, I found that it worked after adding just this line: $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;. So, is that safe to do? Is there a better solution? Thanks. – mepstein May 26 '15 at 02:36
  • First, it is not safe to disable verification unless you take https as something which must be worked around instead of a security feature. Next, LWP::UserAgent (super class of WWW::Mechanize) uses Mozilla::CA as CA store unless you are on Debian/Ubuntu where they patched it to use the system CA store. And finally you probably gave the wrong path to PERL_LWP_SSL_CA_FILE so that it complained that the file does not exist. My setting was just an example, I have no idea where you put the file yourself. – Steffen Ullrich May 26 '15 at 04:17
  • If you do `$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;`, encryption will still be used, but you don't check who you are talking to, so you could be sending your info to an attacker after telling them how to decode your encryption. – ikegami May 26 '15 at 04:17
  • Ahh, OK. Think I got it now. I added the line `$ENV{PERL_LWP_SSL_CA_PATH} = '/etc/ssl/certs';` (and removed the line setting PERL_LWP_SSL_VERIFY_HOSTNAME), and the script works. And I guess at some point Mozilla::CA will be updated, and that will eliminate the need for this line. Thanks. – mepstein May 26 '15 at 05:18
  • Yes, and you will get other problems when Mozilla::CA gets updated because of all the missing 1024 bit certificates, see http://stackoverflow.com/a/30068150/3081018 – Steffen Ullrich May 26 '15 at 05:34