Web Crawler issue in perl

Question

I had built a web crawler in Perl.

I am using

HTML::ContentExtractor

LWP::UserAgent 

HTML::LinkExtor

to extract text form webpages.

Reference link for sample code web cralwer perl

Issue:

The issue is that it does not get text from web pages that have the extension as .aspx . It works perfectly for other webpages.I could not figure out the issue why this crawler fails for aspx pages.

javascript is not supported. you have to create POST or get requests — run, Apr 25 '14 at 07:06
Can you suggest me how to do that?I had tried $ua->post( $url, \%form ) $ua->post( $url, \@form ) but of no use.@run — Nagaraju, Apr 25 '14 at 09:16

score 1 · Answer 1 · answered Apr 25 '14 at 07:04

1

To be able to access a process a website with JavaScript, use WWW::Mechanize::Firefox

answered Apr 25 '14 at 07:04

Miller

34,962
4
39
60

Web Crawler issue in perl

1 Answers1