Scripts broke after upgrading LWP "certificate verify failed"

Question

I have a lot of scripts, most of them based around WWW::Mechanize that scrape data off of misc hardware that is accessible via HTTPs. After upgrading most of my perl installation and its modules, all scripts using HTTPS:// broke because of "certificate verify failed"

This is a result of the fact that the newer versions of LWP does a proper check on the certificate and dies if something doesn't match.

In my case, the failed certificate authentication is expected due to the circumstances, so i needed to find a way of cleanly circumventing this check.

ikegami · Answer 1 · 2016-10-31T19:48:18.440

Say I want to tell you something, and I don't want anyone else to know it. We'd arrange a password, and I'd use it to encrypt the message, then I'd send you the message.

What if I didn't make sure the person to whom I gave the password and encrypted message was you? Then any number of people could simply impersonate you and the encryption would be for naught. That was the state of LWP's HTTPS support until recently.

Now, LWP actually checks to whom it's talking unless you ask LWP to behave as it once did. You can do that using:

my $ua = LWP::UserAgent->new(
   ssl_opts => { verify_hostname => 0 },
);

If you want to affect all LWP::UserAgent instances in your script without specifying the option all over the place, you can add the following to your script

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;

Or you can launch your script as follows:

PERL_LWP_SSL_VERIFY_HOSTNAME=0 script.pl

Finally, if you want LWP to always be unsafe, you can add the following to your login script:

export PERL_LWP_SSL_VERIFY_HOSTNAME=0

However, I recommend none of the above. The far better option would be to provide the certificate for the host to which you are communicating. (This is the equivalent of adding an exception in Firefox, if you know what I mean.) See the documentation for $ua->ssl_opts.

It's a closed network, and all hosts are trusted by default. If they weren't, I'd have bigger issues on my hands. — Jarmund, Apr 07 '15 at 19:43

score 15 · Answer 2 · edited Jul 18 '13 at 21:13

15

For me, using:

my $ua = LWP::UserAgent->new(
  ssl_opts => { verify_hostname => 0 },
);

Yielded

Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER together with SSL_ca_file|SSL_ca_path for verification. If you really don't want to verify the certificate and keep the connection open to Man-In-The-Middle attacks please set SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.

Using this did not give any warnings:

my $ua = LWP::UserAgent->new(
  ssl_opts => { SSL_verify_mode => 'SSL_VERIFY_NONE'},
);

edited Jul 18 '13 at 21:13

Yang

7,712
9
48
65

answered Jul 18 '13 at 20:20

user2597132

159
1
2

1

Thanks, this worked for me. You can also do `my $mech = WWW::Mechanize->new( ssl_opts => { SSL_verify_mode => 'SSL_VERIFY_NONE' } );` – Will Sheppard Nov 21 '14 at 13:54
3

I think you need to have `use IO::Socket::SSL;` and also omit the quotes -- IE, `{ SSL_verify_mode => SSL_VERIFY_NONE }` – Steve Shipway Aug 29 '16 at 00:45
2

Don't use this. `SSL_verify_mode` doesn't want a string. `SSL_verify_mode => 'SSL_VERIFY_NONE'` is probably being treated as `SSL_verify_mode => 0` – ikegami Mar 08 '19 at 21:22

score 8 · Accepted Answer · edited Mar 08 '19 at 21:24

8

I prepended my code with:

$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;

This caused the script to circumvent the check in a clean and simple way.

edited Mar 08 '19 at 21:24

ikegami

367,544
15
269
518

answered Nov 14 '12 at 19:24

Jarmund

3,003
4
22
45

@ikegami OOps, i must've done an export in the shell and forgotten about it, then, as it worked. Regardless, if it had done something, it still wouldn't be preferrable, but i guess this calls for some editing. – Jarmund Nov 14 '12 at 19:28
If you're using this in Bash, type in your terminal `SET PERL LWP SSL VERIFY HOSTNAME = FALSE` then you can proceed again with your precious time, while perl continues with his weird thing. – ReneFroger Jun 28 '19 at 23:57

score 5 · Answer 4 · answered Jan 09 '15 at 20:48

@ikegami makes a good argument for why you don't want to disable SSL hostname verification, but doesn't directly mention how to avoid it.

If you're talking to a public system with a CA-signed certificate, you need to point LWP to your distribution's root certificate collection. Under a Debian-based system (Ubuntu, etc.), this is kept under /etc/ssl/certs/.

BEGIN {
    $ENV{HTTPS_CA_DIR} = '/etc/ssl/certs'
}

If you are talking to your own server with a self-signed certificate, you can save a copy of that certificate on the client, and point your script to that particular file.

BEGIN {
    $ENV{HTTPS_CA_FILE} = '/path/to/my/server-certificate.crt'
}

You could instead set these in the environment before running your script (e.g. export them from your shell), or you could apply the settings directly to your UserAgent object. See the LWP::UserAgent documentation for more details; search for ssl_opts (around halfway down the page).

It's a closed network, and all hosts are trusted by default. If they weren't, I would have bigger problems to worry about :) — Jarmund, Jan 10 '15 at 21:54
Your question helped me solve a similar problem -- thanks! -- but since my network isn't quite as trusted as yours, I wanted to make sure there was an easy answer posted that didn't involve disabling security features. — Jander, Jan 10 '15 at 22:47

score 0 · Answer 5 · answered Jun 28 '22 at 12:37

Add the library in the top, and add rest of the ENV declarations and IO socket info before using the LWP module functions. Then try, It will workout.

use IO::Socket::SSL;
use LWP::UserAgent;

$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
$ENV{HTTPS_DEBUG} = 1;
        
IO::Socket::SSL::set_ctx_defaults(
  SSL_verifycn_scheme => 'www',
  SSL_verify_mode => 0,
);

$ua = LWP::UserAgent->new();

Scripts broke after upgrading LWP "certificate verify failed"

5 Answers5

Linked