How to fail over

Question

If I use wget to retrieve something from the geonames.org server, it reports two IP addresses, and the first one fails but it gets it from the second:

Resolving ws.geonames.org (ws.geonames.org)... 5.9.41.208, 176.9.107.169
Connecting to ws.geonames.org (ws.geonames.org)|5.9.41.208|:80... failed: Connection refused.
Connecting to ws.geonames.org (ws.geonames.org)|176.9.107.169|:80... connected.
HTTP request sent, awaiting response... 200 OK

But unfortunately I have to access it through perl using LWP::UserAgent and HTTP::Request. How can I make them try the second IP if the first fails?

    my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
    GET =>
      "http://ws.geonames.org/countrySubdivision?lat=$lat&lng=$long&radius=$radius&username=xyzzy");

my $res = $ua->request($req);

Are you sure that this isn't the default behaviour for LWP::UserAgent anyway? After digging through the LWP::UserAgent code today, it would appear that when the stream socket is being connected deep down inside IO::Socket::INET::configure() it will try all IP addresses returned for a given hostname and will settle for the first IP that it can connect to. — Rob Wells, Jun 16 '14 at 19:58
@RobWells, the problem was that there is a difference between "can open a socket to" and "get a valid HTTP response within the timeout time". `wget` would go on to the next one if the second part failed, but `LWP::UserAgent` does not. — Paul Tomblin, Jun 16 '14 at 20:04
cheers @Paul. I've seen that you have to explicitly enable the MultiHome option to get that behaviour to loop across multiple IP's. It's buried in a morass of Perl OO ->SUPER::foo so it's "fun" to track down! (-: — Rob Wells, Jun 17 '14 at 15:35
It appears it's not easy to set: https://rt.cpan.org/Public/Bug/Display.html?id=60104 and that also implies it might not do what I want. I'll have to try. — Paul Tomblin, Jun 17 '14 at 16:46

score 6 · Accepted Answer · answered Aug 24 '13 at 23:27

You can do it yourself: get all the IP addresses with the help of Net::DNS::Resolver, and then try all IP addresses until you get a successful response. Note that you have to supply the "Host" header yourself if working with an IP address, in case the server is doing name-based virtual hosts.

Something like the following lines could work. Maybe there's even a CPAN module for this, I did not check:

use Net::DNS;
use LWP::UserAgent;

my @addrs;
{
    my $res   = Net::DNS::Resolver->new;
    my $query = $res->search("ws.geonames.org");
    if ($query) {
        for my $rr ($query->answer) {
            if ($rr->type eq "A") {
                push @addrs, $rr->address;
            }
        }
    } else {
        die "DNS query failed: ", $res->errorstring, "\n";
    }
}

my $ua = LWP::UserAgent->new;

my $res;
for my $addr (@addrs) {
    $res = $ua->get("http://$addr/countrySubdivision?lat=$lat&lng=$long&radius=$radius&username=xyzzy", Host => 'ws.geonames.org');
    last if $res->is_success;
}

sorry, did indeed not see that before writing the comment. I've therefore deleted the comment as it doesn't apply at all. — Moritz Bunkus, Aug 25 '13 at 08:44

score 1 · Answer 2 · answered Apr 21 '15 at 15:00

The solution from Slaven is OK except when the IP addresses are not directly accessible. In that case, the following works for me:

local @LWP::Protocol::http::EXTRA_SOCK_OPTS = (
                                           PeerAddr   => 'my_hostname',
                                           MultiHomed => 1,
                                          );
my $response = $ua->post('https://my_hostname/...', ...);

How to fail over

2 Answers2