8

My symptom is that I cannot use a proxy with HTTPS requests with LWP. This seems to be a common problem, and the hints on Google and even here all suggest a work-around for setting the HTTPS_PROXY environment variable for use by Crypt::SSLeay.

My specific problem appears to be that LWP::Protocol::https is loading IO::Socket::SSL rather than Crypt::SSLeay. How can I force Crypt::SSLeay's use instead?

My code:

#!/usr/bin/perl

use strict;
use warnings;
$ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://www.meritrustcu.org/');
my $res = $ua->request($req);
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;

And it's output, showing that Crypt::SSLeay is not being used:

Net/SSLeay.pm
IO/Socket/SSL.pm
/usr/lib/perl5/auto/Net/SSLeay/autosplit.ix
/usr/lib/perl5/auto/Net/SSLeay/set_proxy.al
/usr/lib/perl5/auto/Net/SSLeay/randomize.al

Simply adding an explicit use Crypt::SSLeay to my script has proven ineffective. It loads the module, but it continues to load IO::Socket::SSL, and use it for the HTTPS requests.

Community
  • 1
  • 1
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189

2 Answers2

9

Try this:

use strict;
use warnings;

use Net::SSL (); # From Crypt-SSLeay
BEGIN {
  $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
  $ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';
}

use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://www.meritrustcu.org/');
my $res = $ua->request($req);
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;

I don't have a suitable proxy, so I haven't tried it myself.

cjm
  • 61,471
  • 9
  • 126
  • 175
  • 1
    Perfect, thanks! Doing this revealed the likely reason that LWP defaults to IO::Socket::SSL, as you touched on in your [own question](http://stackoverflow.com/questions/74358/how-can-i-get-lwp-to-validate-ssl-server-certificates): `Net::SSL from Crypt-SSLeay can't verify hostnames; either install IO::Socket::SSL or turn off verification by setting the PERL_LWP_SSL_VERIFY_HOSTNAME environment variable to 0` – Jonathan Hall Nov 06 '11 at 11:13
  • 1
    LWP 6 changed it's policy on validating host names. If you can, use something other than Crypt::SSLeay, which doesn't have a dedicated developer (although Sinan has been helping it compile). – brian d foy Nov 06 '11 at 12:30
  • @briandfoy: If I can figure out how to use IO::Socket::SSL (or another option--are there other options?) with a proxy, I'll do that. – Jonathan Hall Nov 07 '11 at 04:56
  • @cjm so in the code you are setting `$ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';` that's not a `secured` proxy. It should be `$ENV{HTTPS_PROXY} = 'https://10.0.3.1:3128';` ?????? isn't that right? – Anjan Biswas Aug 24 '12 at 21:49
  • @Annjawn, well that depends on whether your proxy listens via https. I just copied the proxy setting from the question. – cjm Aug 24 '12 at 22:20
  • OK, I thought `$ENV{HTTPS_PROXY}` is for `https` proxies and `$ENV{HTTP_PROXY}` is for usual `http` proxies. – Anjan Biswas Aug 24 '12 at 22:35
1

This is what I did to get LWP and SOAP::Lite to work with our proxy at GE. This was after much digging on CPAN, google etc. I finally figured it out after running the test script in the Crypt::SSLeay package called net_ssl_test and it was able to connect through the proxy. The key is forcing Crypt::SSLeay to use Net::SSL as was mentioned above. This is not documented very well on CPAN however.

use LWP::UserAgent;

# override HTTPS setting in LWP to work with a proxy
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL";
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_PROXY}               = 'http-proxy.ae.ge.com:80';
$ENV{HTTPS_PROXY_USERNAME}      = 'username';
$ENV{HTTPS_PROXY_PASSWORD}      = 'password';

$ua = new LWP::UserAgent;

# make a https request
my $req = HTTP::Request->new(GET => 'https://mail.google.com/');
my $res = $ua->request($req);
print $res->as_string;
Rich Wellman
  • 36
  • 1