Perl HTTPS over proxy using LWP::UserAgent

Question

I wish to request a URL via a HTTPS proxy using perl's LWP::UserAgent module. There is quite a few reference around this, but nothing could help me get it work.

#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use Data::Dumper;

BEGIN {
    $ENV{HTTPS_PROXY} = 'https://<IP>:<PORT>';
    $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
    $ENV{HTTPS_PROXY_USERNAME} = '<API_KEY>';
    $ENV{HTTPS_PROXY_PASSWORD} = '';
    $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

my $ua = LWP::UserAgent->new(ssl_opts => {verify_hostname => 0}, SSL_version => 'SSLv3', allowed_protocols => ['https', 'http']);
$ua->proxy(['https', 'http'], 'https://<IP>:<PORT>');
my $req = HTTP::Request->new('GET','https://<DOMAIN_URL>');
print STDERR Dumper($ua);
my $response = $ua->request($req);
print $response->code ."\n";
print STDERR Dumper($response);

I get this error:

SSL connect attempt failed error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol at /home/user/project/local/lib/perl5/LWP/Protocol/http.pm line 51.

Despite me specifying allowed_protocols in LWP, https scheme in proxy and the url scheme being https, it still goes to LWP::Protocol::HTTP above and not to LWP::Protocol::HTTPS.

I also verified that the version of LWP::Protocol::HTTPS is 6.06 which is the same as LWP::UserAgent (which was mentioned in one of the forums)

Not really I'm afraid. If you can provide a server to test with, I can play around. But off the top of my head I have no ideas. — simbabque, Sep 06 '16 at 10:41
Why are you setting `SSLv3`? Is that because you really need it, or because you saw it somewhere? — Sinan Ünür, Sep 06 '16 at 13:59

score 3 · Answer 1 · edited May 23 '17 at 12:16

3

This worked for me:

use LWP::UserAgent;

$ua = LWP::UserAgent->new(); 
$ua->proxy('https', 'connect://<USER>:<PSWD>@<IP>:<PORT>/');

$ua->get('https://www.somesslsite.com');

HTTPS Proxy and LWP::UserAgent

Note: The environment credentials (HTTPS_PROXY_USERNAME) didnt work for me. I had to enter it in the URL like above.

edited May 23 '17 at 12:16

Community

1
1

answered Sep 06 '16 at 10:52

alpha_cod

1,933
5
25
43

1

The `connect://` scheme works only if you have `LWP::Protocol::connect` installed. It works also with version of LWP before 6.06. But with versions starting with 6.06 I recommend to use the builtin proxy facility because you don't have to install an additional modul. – Steffen Ullrich Sep 06 '16 at 14:44

score 2 · Answer 2 · answered Sep 06 '16 at 13:05

The correct setup with recent versions of LWP (starting with version 6.06 which you have) is to just use the same syntax as found in other applications together with the proxy function:

 my $ua = LWP::UserAgent->new;
 $ua->proxy(https => 'http://user:pass@proxy');
 $ua->get('https://server');

Alternatively you could set the environment variable https_proxy to the same value, i.e. http://user:pass@proxy.

Before 6.06 proxy support was broken at least when used together with IO::Socket::SSL (default since version 6.0). The syntax you have in your question is for the old backend Crypt::SSLeay which is no longer recommended because it does not check the certificates properly.

Agreed. OP, please see [Do you need Crypt::SSLeay?](https://metacpan.org/pod/Crypt::SSLeay#DO-YOU-NEED-Crypt::SSLeay) and [Does your code really depend on Crypt::SSLeay?](https://www.nu42.com/2014/04/does-your-code-really-depend-on.html). — Sinan Ünür, Sep 06 '16 at 14:04

Perl HTTPS over proxy using LWP::UserAgent

2 Answers2