perl: extending code to get particular information from website

Question

I have an ID code representative of a protein. There is website called InterPro that is used to deal with proteins related information. URL for that website contains that particular code. By changing that code in that URL I can get information about any protein. I developed a script in perl to get information directly from web. I used following code

    my $uniprot= "P15700";
    my $resp= '';
    my $url= "http://wwwdev.ebi.ac.uk/interpro/ISearch?query=$uniprot+";
    my $file = "$uniprot";
    $resp = getstore( $url, $file ); 


exit;
}

In this example P15700 is that unique ID of protein and the url is http://wwwdev.ebi.ac.uk/interpro/ISearch?query=P15700+. Now this will retrieve whole html page. But, I need a particular information on that page which under the heading of "protein family membership". Like in this example If you open this link you can find "Adenylate kinase" written under heading of protein family membership. I just neeed to that information in another output text file displaying a table in which one column says ID and other says that information under protein family membership. I am new to perl and I don't have computer science background. Rather I am a biologist. So, I want to know whether the above mentioned task can be done using perl. If yes, How? I ll be grateful if anyone can solve this problem.

Answer 1

use strictures;
use Web::Query 'wq';
my $w = wq 'http://wwwdev.ebi.ac.uk/interpro/ISearch?query=P15700+';
$w->find('.prot_fam a')->text;
# expression returns:
# (
#     'Adenylate kinase',
#     'UMP-CMP kinase',
# )

Answer 2

This relates to parsing web page HTML which IMO is rarely a good idea. The page may change at any time and that will cause you script to stop working properly. If you are still interested here's the solution:

use Mojo::DOM;    
my $dom = Mojo::DOM->new($resp);
my $name = $dom->find('div.prot_fam a')->[0]->text;

Now $name variable will hold the Adenylate kinase string.

Answer 3

Everything can be done using Perl :) As for this particular problem, take a look at this question of mine concerning recursive web download and DOM code.

As you're not a programmer, much of this will be news to you.

Let's understand the DOM first. It's the HTML tree built in the browser when viewing web pages. You can acquire a decent understanding of the DOM playing around with Firebug or the equivalent plug-ins or built-ins for Chrome and IE and Opera, whichever one you're using.

So you will have to go to this page and analyze its DOM. It looks like the info you're looking for is in a <div class="prot_fam"> element. So that's all the info you need to write the code:

D:\ :: more /t2 prot.pl
use strict;
use warnings;
use LWP::UserAgent;
use HTML::TreeBuilder::XPath;

my $url  = shift || die 'pass URL as argument!';
my $file = shift || die 'pass output filename as argument!';

my $ua = LWP::UserAgent->new;
my $rsp = $ua->mirror( $url, $file );
if ( ! $rsp->is_success ) {
  die $rsp->status_line;
}

my $tree = HTML::TreeBuilder::XPath->new;
$tree->parse_file( $file ) or die;

print $_, "\n" for map $_->as_XML_indented,
$tree->findnodes(q( //div[@class='prot_fam'] ));

D:\ :: perl prot.pl http://wwwdev.ebi.ac.uk/interpro/ISearch?query=P15700 P15700.html
<div class="prot_fam">
  <div class="entry-parent">
    <div class="entry-parent">
     <a href="IEntrySummary?ac=IPR000850&amp;query=P15700">Adenylate kinase</a>
      <div class="entry-child-prot">
        <div class="entry-parent">
         <a href="IEntrySummary?ac=IPR006266&amp;query=P15700">UMP-CMP kinase</a>
        </div>
      </div>
    </div>
  </div>
</div>

Adding another sample using Mojo::DOM

use strict;
use warnings;
use LWP::UserAgent;
use Mojo::DOM;
my $url = shift || die 'URL!';
my $ua  = LWP::UserAgent->new;
my $rsp = $ua->get( $url );
my $dom = Mojo::DOM->new($rsp->content);
for ( $dom->find('div[class="prot_fam"]')->each ) {
    print $_->find('a'), "\n";
}

Answer 4

Not even sexy, but it works (based on HTML::Treebuilder Module) - you have to parse the HTML and extract the information. In this example the result will be stored as csv in the File "result.txt"

use LWP::Simple;
use HTML::TreeBuilder;

my $uniprot= "P15700";
my $url= "http://wwwdev.ebi.ac.uk/interpro/ISearch?query=$uniprot+";
my $resp = get( $url );

my $tree = HTML::TreeBuilder->new_from_content($resp);
my $first=$tree->look_down(_tag => 'div',class => 'prot_fam') ;
$first=$first->look_down(_tag => 'div',class => 'entry-parent');
$first=$first->look_down(_tag => 'div',class => 'entry-parent');
$first=$first->look_down(_tag => 'a');
open (FH,">>result.txt");
print FH $uniprot.";";
print FH $first->content_list;
print FH "\n";
close(FH);

Edit: Here's a variant for checking lot's of "uniprots". Play around with the sleepdelay

use LWP::Simple;
use HTML::TreeBuilder;

my @ports=qw(Q9H4B7 Q96RI1 P04150 P35354 P23219 P61073 P0A3M6 Q8DR59 Q7CRA4 Q27738 P35367 P35367 P35367 P08172 P35367 P10275 P25021 P07550 P08588 P13945);

for (my $i=0;$i < scalar(@ports);$i++) {
my $url= "http://wwwdev.ebi.ac.uk/interpro/ISearch?query=".$ports[$i]."+";
my $resp = get( $url );

my $tree = HTML::TreeBuilder->new_from_content($resp);
my $first=$tree->look_down(_tag => 'div',class => 'prot_fam') ;
$first=$first->look_down(_tag => 'div',class => 'entry-parent');
$first=$first->look_down(_tag => 'div',class => 'entry-parent');
$first=$first->look_down(_tag => 'a');
open (FH,">>result.txt");
print FH $ports[$i].";";
print FH $first->content_list;
print FH "\n";
close(FH);
sleep 10;
}