HTTPS Proxy and LWP::UserAgent

Question

I have read a number of threads on a number of sites and am still unable to make this work.

I have a client machine (OSX) with OpenSSL 0.9.8r running perl 5.12.4, with LWP 6.0.4, updated Crypt::SSLeay, Net::SSL etc. I am trying to connect to an HTTPS site (https://github.com in the example) via a WinGate proxy that I have running on a Windows VM. Note that my actual application is attaching to an SSL webservice that I have no control over.

From Firefox, pointed to the proxy everything is copacetic. The page loads successfully and I see the connections in the Proxy software Activity monitor. I'll be darned if I can make it work in Perl though. I've started with the code from this Stack Overflow Question : How do I force LWP to use Crypt::SSLeay for HTTPS requests? And added some debugging and additional output. Here's were I stand now:

#!/usr/bin/perl

use strict;
use warnings;
use Net::SSL (); # From Crypt-SSLeay

BEGIN {
  $Net::HTTPS::HTTPS_SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
  $ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
#  $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
  $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);

print "--\n";
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
print "--\n";

if ($response->is_success) {
     print $response->decoded_content;  # or whatever
     exit(0);
}
else {
 print "\nFail:\n";
     print $response->status_line ."\n";
     exit(1);
}

Here's the output from this code:

--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
--

Fail:
500 Can't connect to github.com:443 (Crypt-SSLeay can't verify hostnames)

If I then uncomment $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;, I do see a single connect to github.com:443 on the proxy and then nothing. (Note it works great from a web browser through the proxy). After much hanging I get the following output from the script:

SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:failed in SSLv2 read server hello A
--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
Crypt/SSLeay/CTX.pm
Crypt/SSLeay/MainContext.pm
--

Fail:
500 SSL negotiation failed:

If anyone can provide some direction here I would greatly appreciate it!

Why do you want to `Force use of Net::SSL`? I gthink `LWP` should take care of the `SSL` nego itself. You just have to point it to a correct proxy. — Anjan Biswas, Aug 24 '12 at 21:14
Note that your use of `HTTPS_SSL_SOCKET_CLASS` doesn't actually do anything (you did not copy the answer correctly), but that this also doesn't seem to matter since your system defaults to `Net::SSL`. The script looks fine, so there is some other variable you're not accounting for. I don't know enough about the libraries in play to provide a real answer, but variables I would consider include the network (is the script on the same box as the browser?), configuration (do you need a username or password?), and interactivity (is the proxy's certificate good)? — darch, Aug 25 '12 at 04:39
darch, Thanks for the comment. I have updated the script to leverage the `$NET::HTTPS::SSL_SOCKET_CLASS` package variable as per the original answer. I have seen the environment variable used in a number of other examples. Yes the browser is on the same machine as the script, no username/password are required, as per the certificate the only thing I can suggest is that the browser does not have any issue with it. Thanks again for replying. Any help is welcome. — Ted B, Aug 27 '12 at 21:24

score 7 · Accepted Answer · answered Jul 22 '13 at 12:05

I just uploaded the LWP::Protocol::connect module to CPAN. This module adds the missing HTTP/CONNECT method support to LWP.

  use LWP::UserAgent;

  $ua = LWP::UserAgent->new(); 
  $ua->proxy('https', 'connect://proxyhost.domain:3128/');

  $ua->get('https://www.somesslsite.com');

With this module you can use the regular IO::Socket::SSL implementation for LWP >=6.00.

Anjan Biswas · Answer 2 · 2012-08-24T21:41:47.250

Why would you want "Force use of Net::SSL". Try

#!/usr/bin/perl    
use strict;
use warnings;
use LWP::UserAgent;

BEGIN {
  $ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
#  $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
  $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print $response->code ."\n";

An out put of 200 should mean that there were no errors.

A below sample code of mine works perfectly

#!/usr/bin/perl
use warnings;
use LWP::UserAgent;

BEGIN {
  $ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080'; #Valid HTTPS proxy taken from http://hidemyass.com/proxy-list/
  $ENV{HTTPS_DEBUG} = 1;
}

my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://www.nodeworks.com');
my $res = $ua->request($req);
print $res->code, "\n";

Output-

200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A

Tool completed successfully

With https://github.com/ the output is-

200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A

Tool completed successfully

So having said all this. Your code version (below) should work fine-

use warnings;
use LWP::UserAgent;

BEGIN {
  $ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080';
  $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; #works even with this
  $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://github.com/');
my $res = $ua->request($req);
print $res->code, "\n";

if ($res->is_success) {
     print $res->decoded_content;  # or whatever
     exit(0);
}
else {
 print "\nFail:\n";
     print $res->status_line ."\n";
     exit(1);
}

Annjawn, Thanks so much for taking the time to post an answer. Your modified version of my script does indeed return the github content fine, but it does not ever hit the proxy server. My understanding is that using this "vanilla" LWP approach (which is where I started) you need to call `$ua->env_proxy` in order to leverage the environment variables. When I do this, your example no longer works. If you would be willing to validate that I would appreciate it. Thanks again for responding. — Ted B, Aug 27 '12 at 21:27
Downvoted because not an answer. It's a restatement of the question and editorializing on choices not relevant to his problem. "This different thing works for me." does not answer "Why does this thing not work?". — darch, Aug 30 '12 at 15:41
@darch... I prefer to concentrate on the technicality of the question and the issue rather than the language and grammar. Not sure what you mean by "not relevant to the question" because as far as I can see everything in the answer is relevant to the question technically. — Anjan Biswas, Aug 30 '12 at 15:54

score 1 · Answer 3 · answered Sep 27 '12 at 15:47

I pretty much ran into the same problem. Here are the things that fixed it for me:

make sure you use Crypt::SSLeay 0.58 or lower. At least 0.64 blocked endlessly for me (as described at http://cpanforum.com/posts/13808 ), while 0.57/0.58 worked.
make sure Net::HTTP has been updated to at least 6.03 (to avoid https://rt.cpan.org/Public/Bug/Display.html?id=72790 )

score 1 · Answer 4 · answered Nov 04 '13 at 20:21

Instead of using Net::SSL which does not provide much host verification (and no SNI) you can use Net::SSLGlue::LWP. This monkey-patches LWP so that https_proxy can be used with the default SSL backend IO::Socket::SSL:

use Net::SSLGlue::LWP; # do this first
use LWP::Simple;
... continue with normal LWP stuff..

score 0 · Answer 5 · answered Mar 21 '13 at 13:17

I know this may be a dead question, but if anyone else hits it I've another angle... I can't promise any answers, but we've faced a long-standing problem at $work in this area, but with the Squid proxy, and maybe specific to use of X509 client certs.

The use of the Net::SSL override is part of the solution, but I would fear that WinGate could be the problem (and not something I can help with) although in our case we contact the proxy over http (not sure how LWP deals with proxy+https).

For the record, here's an example of the precise form of code we use:

use Net::SSL;
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS}="Net::SSL";
use LWP::UserAgent;
use LWP::Protocol::https;
my $ua = LWP::UserAgent->new;
$ENV{HTTPS_PROXY} = 'http://cache.local.employer.co.uk:80';
$ua->get("https://example.com/");

This is Perl 5.8.8 with recent CPAN installs (hence separation of L:P:https), so we have a fresh Net::HTTP.

I was going to mention some versions of Net::HTTP are crocked, but I just realised that was my CPAN bug in Martin's reply :)

Sorry if this doesn't add anything.

https over an http proxy just uses the CONNECT method, it's very simple, the proxy processes the CONNECT request (makes connection and sends 200 OK back to the client) and goes into pass-thru mode. All manner of problems can happen if the proxy is implementing policy / blocking this though, as clients typically handle this very poorly. Disclaimer, I work for Qbik who are the authors of WinGate — Adrien, May 30 '16 at 13:38

score 0 · Answer 6 · edited Nov 04 '13 at 02:33

0

I've sent a pull-request on libwww-perl repository to fix (or may be workaround ...) the issue.

The comment of this PR shows a simple program that connects with https to github.com through a proxy. With this patch there's no need to mess around with %ENV in your program.

Another advantage is that you can re-use the usual https_proxy setup.

edited Nov 04 '13 at 02:33

jamesmortensen

33,636
11
99
120

answered Nov 01 '13 at 18:48

user2946000

1

score 0 · Answer 7 · answered Dec 31 '16 at 02:46

There has been a error in earl 5.8 and some further module where environment variable HTTP_PROXY does not appropriately sets the proxy connection.

Your case has a issue where the bug reported is as mentioned here https://bugzilla.redhat.com/show_bug.cgi?id=1094440

Better way to use this is without environment variable and use LWP UserAgent

 `use LWP::UserAgent;
  $ua = LWP::UserAgent->new(); 
  $ua->proxy('https', 'connect://proxyhost.domain:3128/');`

mimvp · Answer 8 · 2017-07-29T16:35:21.550

#!/usr/bin/env perl 
# 
# mimvp.com
# 2017-03-28

use CGI;
use strict;
use LWP::UserAgent;


our %proxy_https = ("https", "connect://173.233.55.118:443");
our $mimvp_url = "https://proxy.mimvp.com/exist.php";

## https
## 1. download LWP-Protocol-connect (wget http://search.cpan.org/CPAN/authors/id/B/BE/BENNING/LWP-Protocol-connect-6.09.tar.gz)
## 2. tar zxvf LWP-Protocol-connect-6.09.tar.gz 
##    cd LWP-Protocol-connect-6.09
##    perl Makefile.PL
##    make
##    sudo make install
sub test_connect {
 my ($url, %proxy) = @_;
 
 print "proxy  : $proxy{'http'}\n";
 print "https  : $proxy{'https'}\n";
 print "socks4 : $proxy{'socks4'}\n";
 print "socks5 : $proxy{'socks5'}\n";
 print "url : $url\n";
 
 my $browser = LWP::UserAgent->new();
 $browser->env_proxy();
 
#  # 设置的代理格式
 $browser->proxy(%proxy);
 $browser->timeout(30);
 $browser->agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36');
 
#  my $req = new HTTP::Request('GET', $url);
#  my $response = $browser->request($req);
 my $response = $browser->get($url);      # 爬取的网址
 my $is_success = $response->is_success();   # 1
 my $content_type = $response->content_type();  # text/html
 my $content = $response->content();     # 网页正文
 my $content_length = length($content);    # 网页正文长度
 
 print "$is_success\n";
 print "$content_type\n";
 print "$content_length\n";
 print "$content\n";
}

test_connect($mimvp_url, %proxy_https);  # https

## perl mimvp-proxy-perl.pl

Please explain how the code you posted answers the question. — axiac, Jul 29 '17 at 16:45
you can directly run the code, use $browser->proxy(%proxy); to set proxy ip:port address — mimvp, Aug 07 '17 at 13:22

HTTPS Proxy and LWP::UserAgent

8 Answers8

Linked