500 Error on LWP::UserAgent part 2

Question

I have been researching problems with HTTPS 500 Error via LWP::UserAgent. I've tried everything in 500 error with LWP:UserAgent to no avail.

I have two Perl libraries on the same server (system and my own). My own library is more recent, but doesn't work.

Perl v 5.10.1
IO::Socket::SSL v 2.022
LWP::UserAgent v 6.15
LWP::Protocol::https v 6.06

All are greater than what is recommended in the referenced post.

When I run this with trace on, I get:

DEBUG: .../IO/Socket/SSL.pm:2688: new ctx 26906416
DEBUG: .../IO/Socket/SSL.pm:605: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:607: socket connected
DEBUG: .../IO/Socket/SSL.pm:629: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:662: using SNI with hostname zscaler.edwardjones.com
DEBUG: .../IO/Socket/SSL.pm:697: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:716: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:742: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:752: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:772: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:735: local error: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:738: fatal SSL error: SSL connect attempt failed
DEBUG: ...erl5/Net/HTTPS.pm:69: ignoring less severe local error 'IO::Socket::INET configuration failed', keep 'SSL connect attempt failed'
DEBUG: .../IO/Socket/SSL.pm:2721: free ctx 26906416 open=26906416
DEBUG: .../IO/Socket/SSL.pm:2726: free ctx 26906416 callback
DEBUG: .../IO/Socket/SSL.pm:2733: OK free ctx 26906416

I used a packet capture and discovered that this uses Secure Socket Layer -> SSL Record Layer. The older, system library uses Secure Socket Layer -> TLSv1 Record Layer. The failing packet is > 254 bytes, but I am using newer Perl code.

I've tried verify_hostname 0 and 1 with no change.
I've tried SSL_Version => TLSv1 with no change.

Here is the code:

#!/usr/bin/perl

use strict;
use warnings;

use lib '/opt/mylib/lib/perl5';

use Data::Dumper;
use LWP::UserAgent;
use HTTP::Request;
use JSON;

use Net::SSLeay;
$Net::SSLeay::trace = 2;
$Net::SSLeay::ssl_version = 10;  #-- force TLSv1

use IO::Socket::SSL;
$IO::Socket::SSL::DEBUG = 3;

use constant {
    TOKEN              => 'xxxx-yyyy-aaaa',
    GET_URI   => 'https://support.myvendor.com',
};

my $useragent = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 }, cookie_jar => {} );
$useragent->proxy ('https', 'https://my.proxy.com:443');
$useragent->ssl_opts( SSL_version => 'TLSv1');

sendRequest ('MyData');

exit 1;

sub sendRequest {
    my ($data) = @_;
    my $request = HTTP::Request->new('POST', GET_URI);
    $request->header('Content-Type' => 'application/json');
    my $json = encode_json {
        'Token' => TOKEN,
        'Data' => $data
    };
    $request->content($json);
    my $response = $useragent->request($request);
    if (!$response->is_success) {
        print Data::Dumper->Dump([$response], [qw($response)]);
        return;
    }

    eval { $json = decode_json $response->content; 1 } or do {
        print Data::Dumper->Dump([$response], [qw($response)]);
        return;
    };

    print Data::Dumper->Dump([$json], [qw($json)]);
    return;
}

I cannot figure this out. Please help

UPDATE on 6/2/16:

If I set SSL_hostname = "", this is what I get:

DEBUG: .../IO/Socket/SSL.pm:2688: new ctx 38074176
DEBUG: .../IO/Socket/SSL.pm:605: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:607: socket connected
DEBUG: .../IO/Socket/SSL.pm:629: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:665: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:697: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:716: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:742: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:752: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:772: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:2589: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2542: ok=1 [2] /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
DEBUG: .../IO/Socket/SSL.pm:2542: ok=1 [1] /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
DEBUG: .../IO/Socket/SSL.pm:2542: ok=1 [0] /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA/C=US/ST=California/L=San Jose/O=Zscaler, Inc./CN=*.zscaler.net
DEBUG: .../IO/Socket/SSL.pm:2543: local error: Cannot determine peer hostname for verification
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:735: SSL connect attempt failed

DEBUG: .../IO/Socket/SSL.pm:735: ignoring less severe local error 'SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed', keep 'Cannot determine peer hostname for verification'
DEBUG: .../IO/Socket/SSL.pm:738: fatal SSL error: Cannot determine peer hostname for verification
DEBUG: ...erl5/Net/HTTPS.pm:69: ignoring less severe local error 'IO::Socket::INET configuration failed', keep 'Cannot determine peer hostname for verification'
DEBUG: .../IO/Socket/SSL.pm:2721: free ctx 38074176 open=38074176
DEBUG: .../IO/Socket/SSL.pm:2726: free ctx 38074176 callback
DEBUG: .../IO/Socket/SSL.pm:2733: OK free ctx 38074176

If I don't set SSL_hostname, this what I get. Note that the SNI hostname is my proxy - is that right? Shouldn't it be the end host?

DEBUG: .../IO/Socket/SSL.pm:2688: new ctx 30971328
DEBUG: .../IO/Socket/SSL.pm:605: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:607: socket connected
DEBUG: .../IO/Socket/SSL.pm:629: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:662: using SNI with hostname my.proxy.com
DEBUG: .../IO/Socket/SSL.pm:697: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:716: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:742: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:752: waiting for fd to become ready: SSL wants a read first

After more research, I discovered I had verify_hostname => 1 when I set SSL_hostname = ''. I tried with verify = 0 and got a little further. I think this means I was able to get through my proxy and communicate with the end server? It still doesn't work like the old library though.

DEBUG: .../IO/Socket/SSL.pm:2688: new ctx 44513104
DEBUG: .../IO/Socket/SSL.pm:605: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:607: socket connected
DEBUG: .../IO/Socket/SSL.pm:629: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:665: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:716: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:742: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:752: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:772: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:742: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:752: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:772: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:729: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:732: done Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:787: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2721: free ctx 44513104 open=44513104
DEBUG: .../IO/Socket/SSL.pm:2733: OK free ctx 44513104
$response = bless( {
                     '_content' => 'Server closed connection without sending any data back at /opt/ejvfortiprov/lib/perl5/Net/HTTP/Methods.pm line 399.
',
                     '_rc' => 500,

The current debug information do not provide much details except that the SSL handshake fails. Could you please provide the packet captures for both the good and the bad connection (the full capture at cloudshark.org, not just some text from tcpdump) and also the debug output and specifications of the system which works for comparison? Of relevance might also be the version of Net::SSLeay and openssl used by the systems, i.e. `$Net::SSLeay::VERSION` and `Net::SSLeay::OPENSSL_VERSION_NUMBER()` on both systems. — Steffen Ullrich, May 25 '16 at 04:53
Fails ~~~~ Perl v 5.10.1 IO::Socket::SSL v 2.022 LWP::UserAgent v 6.15 LWP::Protocol v 6.15 LWP::Protocol::https v 6.06 $Net::HTTPS::SSL_SOCKET_CLASS = IO::Socket::SSL IO::Socket::SSL = v 2.022 $Net::SSLeay::VERSION = v 1.72 Net::SSLeay::OPENSSL_VERSION_NUMBER() = 268439647 Works ~~~~~~~~ Perl v 5.10.1 IO::Socket::SSL v 1.31 LWP::UserAgent v 5.833 LWP::Protocol v 5.829 LWP::Protocol::https (not in https.pm) $Net::HTTPS::SSL_SOCKET_CLASS = IO::Socket::SSL IO::Socket::SSL = v 1.31 $Net::SSLeay::VERSION = v 1.35 Net::SSLeay::OPENSSL_VERSION_NUMBER() = 268435459 — MWF, May 31 '16 at 22:02
DEBUG: .../IO/Socket/SSL.pm:1471: new ctx 37998400 DEBUG: .../IO/Socket/SSL.pm:332: socket not yet connected DEBUG: .../IO/Socket/SSL.pm:334: socket connected DEBUG: .../IO/Socket/SSL.pm:347: ssl handshake not started DEBUG: .../IO/Socket/SSL.pm:390: Net::SSLeay::connect -> 1 DEBUG: .../IO/Socket/SSL.pm:445: ssl handshake done DEBUG: .../IO/Socket/SSL.pm:1507: free ctx 37998400 open=37998400 DEBUG: .../IO/Socket/SSL.pm:1515: OK free ctx 37998400 — MWF, May 31 '16 at 22:03
I can't upload to cloudshark.org. If there is a way to attach screen shot, I can show you the Secure Sockets Layer output for each. — MWF, May 31 '16 at 22:04
One more thing - I tried the openssl test in the other post and it worked without the server name and failed with it. — MWF, May 31 '16 at 22:08
the last comment is very interesting. The old version you use does not attempt to use SNI while the new version does by default. If the connection fails when using SNI this might indicate a very buggy server or that you are trying to access the server with the wrong name. To work around it try to set `SSL_hostname=''` in `ssl_opts`. This should disable SNI unless LWP resets it somewhere. Have you tried to access the server with the same name using a recent web browser, because they all use SNI too? — Steffen Ullrich, Jun 01 '16 at 04:59
I updated the original posting with info on SSL_hostname. I did put the URL in an IE11 browser and it seemed OK, but I can't pass the required credentials. I think this has something to do with my cloud proxy. — MWF, Jun 02 '16 at 15:28
After more research, I discovered I had verify_hostname => 1 when I set SSL_hostname = ''. I tried with verify = 0 and got a little further. See me updates in the original post above. — MWF, Jun 02 '16 at 15:37
I think LWP simply does not really support proxies which gets accessed by https. This worked with older LWP versions only because it did not verify the certificates properly. — Steffen Ullrich, Jun 03 '16 at 14:26
Thank you Steffen for all the help on this. I had someone try my Perl Lib without a proxy and it worked, so that confirms your statement. Is there a way to turn off certificate checking in LWP? I can do this in 'wget' and it works with the proxy. — MWF, Jun 06 '16 at 14:06
I really don't recommend to switch off certificate validation, but if you want you will [find lots of information about this](https://www.google.com/search?q=lwp+disable+certificate+verification) on the internet. — Steffen Ullrich, Jun 07 '16 at 04:58

score 0 · Answer 1 · answered May 25 '16 at 06:03

0

No solution, but maybe a workaround: Try curl instead of libwww, for example WWW::Curl or LWP::Curl (looks like it has the same interface as LWP::UserAgent)

answered May 25 '16 at 06:03

Sebastian

2,472
1
18
31

500 Error on LWP::UserAgent part 2

1 Answers1