Perl SOAP::Lite interfacing to a vendor API on .NET

Question

Frequent reader, first time question-asker.

I am working on interfacing Perl with a SOAP API provided by one of our vendors (LogRhythm), and seem to have hit a bit of a road-block.

I have been able to interface with the API, and have overcome the Authentication challenges I originally encountered. Now I am having an issue with Name Space tagging in the SOAP Envelope Body when I need to supply parameters to the SOAP call.

The API provided by the Vendor is built around .NET and WCF.

My prototype code currently looks like the following:

use strict;
no strict "refs";

use Data::Dumper;
use IO::Socket::SSL qw (SSL_VERIFY_NONE) ;
use SOAP::Lite;
use Tie::IxHash;

# Username and Password
my $sUID = "<API UserID>";
my $sPWD = "<API Password>";

# WSDL definition URI.  Using a cached local copy using file:/... also works
my $LookupService_wsdl = 'https://melcapi01.soc.ipsec.net.au/LogRhythm.API/Services/LookupServiceBasicAuth.svc?singleWsdl';
my $lrns = 'http://www.logrhythm.com/webservices';

# Ensure that a consistent xmlns:soap value is used, without this we get inconsistent results.
$SOAP::Constants::PREFIX_ENV = 'SOAP-ENV';

# Don't validate SSL Certificate while testing
IO::Socket::SSL::set_defaults(SSL_verify_mode => "SSL_VERIFY_NONE");

# Construct the security header
my %authHash;
tie %authHash, "Tie::IxHash";
%authHash = (
            Username => SOAP::Data->type( '' => $sUID )->prefix('wsse'),
            Password => SOAP::Data->type( '' => $sPWD )->prefix('wsse'),
    );

my $wsse = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd";
my $securityHeader = SOAP::Header->new(
    name   => 'Security',
    uri    => $wsse,
    prefix => 'wsse',
    value  => \SOAP::Data->new(
        name   => 'UsernameToken',
        prefix => 'wsse',
        value  => \%authHash,
      )
);

# Error handling
on_fault => sub { my($soap, $res) = @_; 
    die ref $res ? $res->faultstring : $soap->transport->status;
};

# Define the SOAP Instance
my $lrapi = SOAP::Lite
    -> readable (1)
    -> service($LookupService_wsdl)
    -> on_action( sub {return $action});

# Set the default Namespace
$lrapi->default_ns($lrns);

# Actually get data from the LookupService
my $result;

# Build up the parameters
my @classificationType = ( SOAP::Data->new(name =>'classificationType', value => 2000));

$result = $lrapi->GetClassificationsByType($securityHeader);

print Dumper $result;

This results in the following SOAP Request:

SOAPAction: "http://www.logrhythm.com/webservices/LookupService/GetClassificationsByType"

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope 
    SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" 
    xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:msc="http://schemas.microsoft.com/ws/2005/12/wsdl/contract" 
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" 
    xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" 
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
    xmlns:tns="http://www.logrhythm.com/webservices" 
    xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" 
    xmlns:wsa10="http://www.w3.org/2005/08/addressing" 
    xmlns:wsam="http://www.w3.org/2007/05/addressing/metadata" 
    xmlns:wsap="http://schemas.xmlsoap.org/ws/2004/08/addressing/policy" 
    xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl" 
    xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
    xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" 
    xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" 
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" 
    xmlns:wsx="http://schemas.xmlsoap.org/ws/2004/09/mex" xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <SOAP-ENV:Header>
        <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
            <wsse:UsernameToken>
                <wsse:Username>reporting</wsse:Username>
                <wsse:Password>password</wsse:Password>
            </wsse:UsernameToken>
        </wsse:Security>
    </SOAP-ENV:Header>
    <SOAP-ENV:Body>
        <tns:GetClassificationsByType>
            <classificationType xsi:type="xsd:int">2000</classificationType>
            <classificationTypeSpecified xsi:type="xsd:boolean">true</classificationTypeSpecified>
        </tns:GetClassificationsByType>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Which returns no results. I have done some digging and Request Doctoring using .NET WebService Studio, and have identified that the problem is caused by a missing xmlns identifier on the element in the SOAP Body.

The SOAP Request should look like the following:

SOAPAction: "http://www.logrhythm.com/webservices/LookupService/GetClassificationsByType"

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope 
    SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" 
    xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
    xmlns:msc="http://schemas.microsoft.com/ws/2005/12/wsdl/contract" 
    xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" 
    xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" 
    xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
    xmlns:tns="http://www.logrhythm.com/webservices" 
    xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" 
    xmlns:wsa10="http://www.w3.org/2005/08/addressing" 
    xmlns:wsam="http://www.w3.org/2007/05/addressing/metadata" 
    xmlns:wsap="http://schemas.xmlsoap.org/ws/2004/08/addressing/policy" 
    xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl" 
    xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
    xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" 
    xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" 
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" 
    xmlns:wsx="http://schemas.xmlsoap.org/ws/2004/09/mex" xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <SOAP-ENV:Header>
        <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
            <wsse:UsernameToken>
                <wsse:Username>reporting</wsse:Username>
                <wsse:Password>password</wsse:Password>
            </wsse:UsernameToken>
        </wsse:Security>
    </SOAP-ENV:Header>
    <SOAP-ENV:Body>
        <tns:GetClassificationsByType xmlns="http://www.logrhythm.com/webservices">
            <classificationType xsi:type="xsd:int">2000</classificationType>
            <classificationTypeSpecified xsi:type="xsd:boolean">true</classificationTypeSpecified>
        </tns:GetClassificationsByType>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Various support requests indicate that this should be resolved by manually setting the default_ns on the SOAP::Lite object, however, this is not working for me.

I have gone digging through the source for SOAP::Lite (1.20) to see what is happening, and it seems that the {'_use_default_ns'} property, which is set to 1 before making the SOAP method call, seems to get reset to 0 during processing of the call.

I have also tried to manually build the SOAP::Lite object with the intent of using the call() method to get the desired results, however, I think I am running into issues with the number of name spaces that are auto-added as a result of the service() mechanism I am currently using. If I want to use the call() method it seems I cannot set up the object using service().

Does anyone have any other suggestions I can try as I am at a loss at this point?

Any assistance would be greatly appreciated.

Disclaimer: I am not a programmer/developer, I am a Senior Security Engineer/Consultant/Architect with some programming ability so I apologise in advance if my code is badly structured. This is also my first foray into fiddling with SOAP.

Is this something for a customer? How dirty of a solution would you be comfortable with? Your code doesn't have `use warnings`, but my assessment is that it is not very messy. It's just SOAP::Lite and all the stuff that relates to it that doesn't really look like modern Perl. — simbabque, Sep 01 '16 at 09:14
The LogRhythm API documentation is not just terrible, it is downright misleading. Expect some functions to take arguments in a totally different order to the one documented. — Adam Thompson, Feb 07 '17 at 23:06

score 1 · Accepted Answer · edited May 23 '17 at 10:29

Here is a bit of a hacky solution. It hooks into the Transport layer of your SOAP::Lite client and modifies each outgoing request.

I have used a random web service I found by googling so you can copy and paste the example code and run it without credentials. The service is run by the US National Weather Service. In my example I am using their endpoint to return the lat/long coordinates of a US zip code.

use strict;
use warnings 'all';
use feature 'say';
use SOAP::Lite;

my $client = SOAP::Lite->new(
    proxy => 'http://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php'
);
$client->service('http://graphical.weather.gov/xml/DWMLgen/wsdl/ndfdXML.wsdl');

# the transport layer is a subclass of LWP::UserAgent
$client->transport->add_handler(
    request_prepare => sub {
        my ( $request, $ua, $h ) = @_;
        my $content = $request->content;

        # this is the content before modification
        say 'before: ' . $content;

        # modify the content with a simple regex substitution
        $content =~ s{<LatLonListZipCode>}{<LatLonListZipCode xmlns="http://example.org">};
        $request->content($content);

        # so it doesn't throw a 'Content-Length header value 
        # was wrong, fixed' warning
        $request->header( 'Content-Length' => length $content );

        # this is the content after modification
        say 'after: ' . $content;

        # the return value is ignored
    }
);

my $res = $client->LatLonListZipCode('90210');
say $res->result;

Explanation

This works because the SOAP::Transport layer for this kind of SOAP communication, SOAP::Transport::HTTP::Client, is a subclass of LWP::UserAgent. The UserAgent is very flexible, and offers a lot of different handlers for certain events. The one we are using is request_prepare, which allows us to alter the HTTP::Request before it is sent over the wire (or wireless, or whatever).

The handler is called before the request is sent and can modify the request any way it see fit. This can for instance be used to add certain headers to specific requests.

In the handler that we are adding, we grab the content of the request and use a regex substitute to add an xmlns to the <LatLongListZipCode> tag. While parsing XML with regex is not a good idea, altering a bit of text that happens to look like XML is fine, since we don't care to understand what's in it at that time.

Because the request had already been built, we need to update the header field for Content-Length. It would update automatically, but at the same time issue a warning. We can omit that by simply updating it.

Here is the output of the program, split into three parts.

Content of the request before change

<?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><LatLonListZipCode><c-gensym3 xsi:type="xsd:int">90210</c-gensym3></LatLonListZipCode></soap:Body></soap:Envelope>

Content of the request after change

<?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><LatLonListZipCode xmlns="http://example.org"><c-gensym3 xsi:type="xsd:int">90210</c-gensym3></LatLonListZipCode></soap:Body></soap:Envelope>

Content of the response

<?xml version='1.0'?><dwml version='1.0' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xsi:noNamespaceSchemaLocation='http://graphical.weather.gov/xml/DWMLgen/schema/DWML.xsd'><latLonList>34.0995,-118.414</latLonList></dwml>

Obviously the change worked, the request was still processed and the service I chose doesn't care about that xmlns attribute.

Adopting for your web service

For your use-case, you would have to make sure that only the requests that do not work out of the box get changed. If all of them should be changed, it might be as simple as this. Just do it once after you have built the client.

$lrapi->transport->add_handler(
    request_prepare => sub {
        my ( $request, $ua, $h ) = @_;
        my $content = $request->content;

        # modify the content with a simple regex substitution
        $content =~ s{<tns:GetClassificationsByType>}{<tns:GetClassificationsByType xmlns="http://www.logrhythm.com/webservices">};
        $request->content($content);

        # so it doesn't throw a 'Content-Length header value was wrong, fixed' warning
        $request->header( 'Content-Length' => length $content );
    }
);

Disclaimer

As I said, this is a pragmatic solution. But SOAP is very complex, and there are very few clients that get everything right. There are also a lot of servers that do a lot of things less-than-right. Sometimes pragmatism is the best course of action. Of course the behavior in the server might change, in which case this could break. But that's a risk worth taking I believe.

Thanks for taking the time to put together such a detailed response. I think I understand what you are suggesting, and from my poking around the code, looks like it will work. — John Berkers, Sep 02 '16 at 06:44
You're welcome. The hardest part about this kind of thing in my opinion is to find out where you have to look. I will probably come back here myself at some point for a reference. You don't keep details like this in your head unless you use them every day I guess. :-) — simbabque, Sep 02 '16 at 06:49
The only potential issue is the number of "Methods" that are published by each of several services in the API requiring their request to be doctored, but I have some thoughts about how to deal with that. In any case, I will be sure to update here if this resolves my issue. Thanks again. — John Berkers, Sep 02 '16 at 06:51
You could have a list of tags and possibly the things they need to be updated with. Either build a precompiled regex with `qr` or make a hash and go through that. If there is more than one tag per request, add the `/g` flag to you substitutions. — simbabque, Sep 02 '16 at 06:54

score 1 · Answer 2 · answered Sep 06 '16 at 00:38

As a follow-up to my original post, and to @simbabque, I have tested this and can confirm that @simbabque's suggestion works.

In order to cater for the extended list of available methods (there are 6 distinct service definitions, with a total of 59 methods available. This may increase in future editions of the API...), I have implemented the suggested add_handler as follows:

$lrapi->transport->add_handler(
    request_prepare => sub {
        my ( $request, $ua, $h ) = @_;
        my $content = $request->content;

        # Get the SOAPAction Header from the current request
        my $soapAction = $request->header('SOAPAction');

        # Parse the SOAPAction Header, and determine the method being called
        $soapAction =~ m/^\"(http:\/\/[\w\.\/]+)\/(\w+)\/(\w+)\"$/;
        my ($lruri, $lrsvc, $lrmethod) = ($1, $2, $3);

        # Modify the content with a simple regex substitution
        $content =~ s{<tns:$lrmethod>}{<tns:$lrmethod xmlns="$lruri">};
        $request->content($content);

        # So it doesn't throw a 'Content-Length header value was wrong, fixed' warning
        $request->header( 'Content-Length' => length $content );
    }
);

I also had to set a property on the $lrapi->transport to ensure that it had some values. I simply set the proxy property on the transport to serve this purpose. For some reason the transport property is not built the same way when using a WSDL service resulting in errors.

I used something akin to the following, before calling add_handler, to achieve this:

$lrapi->proxy ("https://melcapi01.soc.ipsec.net.au/LogRhythm.API/Services/LookupServiceBasicAuth.svc");

This results in the $lrapi->transport object being built properly, allowing the add_handler to function correctly.

Thanks again to @simbabque for the solution.

I hope this article is of use to others also.

Glad it helped. You don't need to escape `"` in your regular expression, and if you use a different delimiter (like I did below in the `s{}{}`) you can also get rid of the backslashes in front of the slashes. Inside of the `[]` you only need to escape `]`, while a dot is just a dot, not a meta char. So it could look like this: `m{^"(http://[\w./]+)/(\w+)/(\w+)"$}`. That's way easier to read. — simbabque, Sep 06 '16 at 09:11

Perl SOAP::Lite interfacing to a vendor API on .NET

2 Answers2