Extract InnerHTML from multiple lines in HTML

Question

I am using Perl to connect to a Site, parse its HTML and extract the innerHTML in between the tags. I am trying the easier concept first before trying advanced concepts.

I use LWP::UserAgent to craft my HTTP GET Request to the site and receive my response.

I Store the response in an array as follows:

@res = ($ua->request($req))->content;

Edit: HTML to be parsed:

<div class="new"> this is Line 1 </div>
<div>
      this is Line 2 </div>

Now, I parse each line in the HTTP Response and extract the text between the tags:

foreach $line(@res)
{
chomp $line;
if($line =~ /<div[^>]*?>(.*)<\/div>/)
{
    $match = $1;
    print OUTPUT $match."\n";
}
}

The problems with the above code snippet are:

1. It matches only the innerHTML for the first successful match. It does not print all the successful matches. I am not sure why, the loop should be working according to me. The value of the variable, $match should be overwritten with the contents of capture buffer after every successful match.

2. It will not be able to extract the text between the innerHTML if the tag spans across multiple lines. You have the opening div tag on the first line, innerHTML on the next line and the closing div tag on the following line.

I am unable to write the HTML in this post, so have given the description.

Any help would be appreciated.

Answer 1

Using a robust HTML parser:

use HTML::TreeBuilder::XPath qw();

my $tree = HTML::TreeBuilder::XPath->new;
$tree->parse($http_response->content);

for my $node ($tree->findnodes('//div')) {
    print $_->as_HTML for @{ $node->content_array_ref };
}

Answer 2

You should use progressive matching to extract all matches from a line. For example, if $line holds the string This is a div, followed by a span, and you want to extract This is a div, followed by a and span, you can use something like this:

print "$2\n" while $line =~ /<(.*?)>(.*?)<\/\g{1}>/g;

Of course you want to parse nested elements as well, it's gonna be whole lot more difficult and tricky. As per you second problem, you need multi-line mode. The best would be to use the \s modifier, which will force . to match newline also. Or maybe you can merge all the lines together by assigning the filehandle to a scalar variable directly.

Answer 3

If you want to make it generic enough and suitable for real-world application, it's a bit more complicated.

First, you probably want to get rid of the content between <script> and </script> tags.

Second, you cannot assume that opening tag always contains the same text, e.g. text in <span class="myclass"> is not quite the same as in </span>.

I would suggest getting rid of all <something> tags, regardless of what kind of tag was that, and also removing the <script> tag.

You probably can't get away with just one super-smart regexp, you'd rather use multiple regexps to do the job.

Here is a little script I've put together, works fine on cnn.com (as a sample of non-trivial input). I tried to preserve the line breaks, just to print it nicely, and removed the empty lines -- but obviously, all this might not be necessary.

I also did some dirty trick here, by hiding \n with a dummy \\\\NN string (replacing <script> globally wouldn't work otherwise).

    my $text = "";
    foreach my $line (@res)
    {
        chomp $line;
        $text .= $line . "\\\\NN"; # Hiding the \n's
    }

    $text =~ s/(<script(\s[^<]*)?>.*?<\/script>)//gi;
    $text =~ s/<.*?>/ /g;

    # Beautify it... :)
    $text =~ s/\s{2,}/ /g;
    $text =~ s/\s*\\\\NN\s*/\\\\NN/g;
    $text =~ s/(\\\\NN){2,}/\\\\NN/g;
    $text =~ s/\\\\NN/\n/g;

    print $text."\n";