Perl https proxy problems

Question

I can't seem to get https through a proxy.

Example:

require LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->proxy('https', 'https://proxy:8080');
# $ua->proxy(['https'], 'https://proxy:8080'); # Fails
# $ua->env_proxy; # This also fails.

my $response = $ua->get('https://aws.amazon.com/cloudwatch/');

if ($response->is_success) {
    print $response->decoded_content;  # or whatever
}
else {
    die $response->status_line;
}

Result:

500 Can't connect to aws.amazon.com:443 (timeout) at test.pl line 17.

But if I try the same proxy with curl (also wget) it works just fine.

$ curl --head --proxy https://proxy:8080 https://aws.amazon.com/cloudwatch/
HTTP/1.1 200 Connection established

HTTP/1.1 200 OK
Server: Server
Date: Thu, 08 Dec 2016 16:42:01 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 214187

Perl versions

$ perl -MLWP -le "print(LWP->VERSION)"
6.15
$ perl --version

This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi

I also tried with and without these:

  export HTTPS_VERSION=3 
  export PERL_NET_HTTPS_SSL_SOCKET_CLASS="Net::SSL"
  export PERL_LWP_ENV_PROXY=1 
  export PERL_LWP_SSL_VERIFY_HOSTNAME=0 

My actual goal here is to get aws-scripts-mon working on a machine behind a proxy but it also uses LWP::UserAgent so if I get this working then that will probably also.

Added info

It turns out that if I change to http by $ua->proxy('http', 'http://proxy:8080'); and accesses a http url then it works just fine. The problem is that I need this to work with https.

The error from mon-put-instance-data.pl is:

./mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/

ERROR: Failed to call CloudWatch: HTTP 500. Message: Can't connect to monitoring.eu-west-1.amazonaws.com:443 (timeout)

LWP::Protocol::https::Socket: connect: timeout at /usr/local/share/perl5/LWP/Protocol/http.pm line 47.

Answer 1

Try LWP::Protocol::connect found in https://stackoverflow.com/a/17787133/44620

  use LWP::UserAgent;

  $ua = LWP::UserAgent->new(); 
  $ua->proxy('https', 'connect://proxyhost.domain:3128/');

  $ua->get('https://www.somesslsite.com');

Answer 2

  $ua->proxy('https', 'https://proxy:8080');

LWP does not support the use of HTTP proxies which are accessed using HTTPS. But my guess is that your proxy does not gets access with HTTPS at all, i.e. it gets accessed with HTTP even though it proxies HTTPS requests(*). Thus the code should instead use a http:// URL to access the proxy and not a https:// URL:

   $ua->proxy('https', 'http://proxy:8080/');

Note that this only works in the usual setup, i.e. with IO::Socket::SSL installed on the system and used by LWP. Especially with setting PERL_NET_HTTPS_SSL_SOCKET_CLASS to Net::SSL or explicitly importing Net::SSL into the program the obsolete Crypt::SSLeay will be used where proxy handling is completely different.

(*) Even though the proxy will be accessed by HTTP and not HTTPS the connection is still encrypted. This is done by the client requesting the proxy to create a tunnel to the original target using the CONNECT method and then doing end-to-end SSL inside this tunnel. While there are a some proxies and some clients which support to be accessed by HTTPS too this would essentially mean to build an SSL connection between client and proxy and inside this SSL connection another SSL connection between client and final target, i.e. double encryption.