WWW::Mechanize not handling apostrophes or dashes

Question

I've been working on extracting info from Metacritic, but I've now run into the problem of not being able to extract cleanly text that has apostrophes or dashes.

This problem is illustrated in the following code:

use WWW::Mechanize;
 $reviewspage = 'http://www.metacritic.com/movie/a-band-called-death/critic-reviews';
 $Review = 'In the end Death triumphs, but its allure and obsession remain a mystery.';
 $l = WWW::Mechanize->new();
    $l->get($reviewspage);
    $k = $l->content;
    @Review = $k =~ m{$Review.*?<div class="review_body">(.*?)</div>}s;
    print "@Review\n";

Outputs:

                                Too much of the doc takes our taste for granted; Alice Cooper, Henry Rollins and others wonâ€™t persuade you that Death could have been huge, nor does a clichÃ©d last-act reunion show. But the filmâ€™s alternating inquiry â€” into family love, slow compromise and, yes, death â€” resonates strongly.

Even though the coding on the website is:

<div class="review_body">
                                Too much of the doc takes our taste for granted; Alice Cooper, Henry Rollins and others won’t persuade you that Death could have been huge, nor does a clichéd last-act reunion show. But the film’s alternating inquiry — into family love, slow compromise and, yes, death — resonates strongly.
                            </div>

I've created similar scripts before that have used WWW::Mechanize and none of them have substituted out characters like this.

score 1 · Answer 1 · edited May 23 '17 at 12:05

Metacritic uses utf8 character set:

<meta http-equiv="content-type" content="text/html; charset=UTF-8">

Therefore to print this content to the console, one must adapt to that character set.

On my windows machine, I must run chcp 65001 in the console before executing my perl script. And I must specify the STDOUT character set:

use strict;
use warnings;
use utf8;

use WWW::Mechanize;

binmode STDOUT, ':utf8';   # output should be in UTF-8

my $url = 'http://www.metacritic.com/movie/a-band-called-death/critic-reviews';
my $Review = 'In the end Death triumphs, but its allure and obsession remain a mystery.';

my $lwp = WWW::Mechanize->new();
$lwp->get($url);
my $data = $lwp->content;

if ($data =~ m{$Review.*?<div class="review_body">(.*?)</div>}s) {
    print "$1\n";
} else {
    warn "Review not found";
}

Outputs (newlines added):

Too much of the doc takes our taste for granted; Alice Cooper, Henry Rollins
and others won’t persuade you that Death could have been huge, nor does a
clichéd last-act reunion show. But the film’s alternating inquiry — into
family love, slow compromise and, yes, death — resonates strongly.

score 1 · Answer 2 · edited May 23 '17 at 11:57

Apparently it's a unicode issue.

Per the recommendations in this answer, I was able to get this version of your code to work:

use v5.12 ;
use utf8 ;
use open qw( :encoding(UTF-8) :std ) ;

use WWW::Mechanize ;
 my $reviewspage = 'http://www.metacritic.com/movie/a-band-called-death/critic-reviews' ;
 my $Review = 'In the end Death triumphs, but its allure and obsession remain a mystery.' ;
 my $l = WWW::Mechanize->new() ;
    $l->get($reviewspage) ;   
    my $k = $l->content ;   
    my @Review = $k =~ m{$Review.*?<div class="review_body">(.*?)</div>}s ;
    print "@Review\n" ;

Output:

                                    Too much of the doc takes our taste for granted; Alice Cooper, Henry Rollins and others won’t persuade you that Death could have been huge, nor does a clichéd last-act reunion show. But the film’s alternating inquiry — into family love, slow compromise and, yes, death — resonates strongly.

WWW::Mechanize not handling apostrophes or dashes

2 Answers2