UTF-8 support for RSS parsing

Question

I'm using XML::RSSLite for parsing RSS data I retrieved using LWP. LWP is correctly retrieving in the right encoding but when using RSSLite to parse the data, the encoding seems to be lost and characteres like é, è, à, etc. are deleted from the output. Is there an option to set in order to force the encoding?

Here is my script:

use strict; 
use XML::RSSLite; 
use LWP::UserAgent; 
use HTTP::Headers; 
use utf8; 

my $ua = LWP::UserAgent->new; 
$ua->timeout(10); 
$ua->env_proxy; 
my $URL = "http://www.boursier.com/syndication/rss/news/FR0004031839/FR"; 
my $response = $ua->get($URL); 

if ($response->is_success) { 
   my $content = $response->decoded_content((charset => 'UTF-8')); 
   my %result; 
   parseRSS(\%result, \$content); 
   foreach my $item (@{ $result{items} }) { 
   print "ITEM: $item->{title}\n"; 
   } 
}

I tried to use XML::RSS as it seems to have more option that may be handy in my case but it failed to install unfortunately. :(

If you need help with installing modules, read [What's the easiest way to install a missing Perl module?](http://stackoverflow.com/questions/65865/whats-the-easiest-way-to-install-a-missing-perl-module) and if still necessary, [open a new question](http://stackoverflow.com/questions/ask). — daxim, Apr 23 '12 at 08:41
Many thanks Daxim four your answer, there was a error during copy/past, I have corrected it and the URL is now fine — ehretf, Apr 23 '12 at 10:21
This is not the issue here (as daxim pointed out), but note that most XML parsers require an XML document, which means you need to prevent LWP from attempt to partially parse the XML by using `$response->decoded_content(charset => 'none'); ` — ikegami, Apr 23 '12 at 17:58

score 4 · Answer 1 · edited Apr 24 '12 at 13:37

4

I like that Mojo::UserAgent along with Mojo::DOM already have the support I need without me tracking down the right combinations of modules to use, and it handles the UTF-8 bits without me doing anything special:

use v5.10;
use open qw( :std :utf8 ); 
use Mojo::UserAgent; 

my $ua = Mojo::UserAgent->new; 
my $URL = "http://www.boursier.com/syndication/rss/news/FR0004031839/FR"; 
my $response = $ua->get($URL)->res; 

my @links = $response
    ->dom( 'item > title' )
    ->map( sub { $_->text } )
    ->each;

$" = "\n";
print "@links\n";

I have another example at Painless RSS processing with Mojo

edited Apr 24 '12 at 13:37

answered Apr 23 '12 at 16:04

brian d foy

129,424
31
207
592

I did run it, but it shows broken characters in place of accents. I do have utf-8 terminal, locale and font. This is how output looks like: http://imgur.com/KyVPJ – Apr 24 '12 at 11:10
1

What happens when you run it with `perl -C`? – brian d foy Apr 24 '12 at 13:26
All good - i.e. perl -C q.pl shows correct characters. Further tests showed that -CO/-C2 is enough – Apr 24 '12 at 13:28

score 3 · Accepted Answer · answered Apr 23 '12 at 08:39

3

The RSSLite documentation explicitely states:

Remove characters other than 0-9~!@#$%^&*()-+=a-zA-Z[];',.:"<>?\s

Therefore, the module is hopelessly broken. Try again with XML::Feed

answered Apr 23 '12 at 08:39

daxim

39,270
4
65
132

Thank for you answer, I missed that. I will give a look at XML:Feed. – ehretf Apr 23 '12 at 10:23
Well, the module is broken until someone fixes it. I wouldn't say hopeless, though. – brian d foy Apr 23 '12 at 18:33

UTF-8 support for RSS parsing

2 Answers2