How can I handle Javascript in a Perl web crawler?

Question

I would like to crawl a website, the problem is, that its full of JavaScript things, such as buttons and such that when they are pressed, they do not change the URL, but the data on the page is changed.

Usually I use LWP / Mechanize etc to crawl sites, but neither support JavaScript. any idea?

score 8 · Accepted Answer · answered Sep 22 '10 at 12:55

8

Another option might be Selenium with WWW::Selenium module

answered Sep 22 '10 at 12:55

erickb

6,193
4
24
19

score 7 · Answer 2 · answered Sep 22 '10 at 12:35

7

The WWW::Scripter module has a JavaScript plugin that may be useful. Can't say I've used it myself, however.

answered Sep 22 '10 at 12:35

ishnid

539
2
5

score 5 · Answer 3 · answered Sep 22 '10 at 15:40

5

WWW::Mechanize::Firefox might be of use. that way you can have Firefox handle the complex JavaScript issues and then extract the resultant html.

answered Sep 22 '10 at 15:40

Eric Strom

39,821
2
80
152

score 1 · Answer 4 · answered Nov 18 '13 at 19:47

1

I would suggest HtmlUnit and Perl wrapper: WWW::HtmlUnit.

answered Nov 18 '13 at 19:47

Minh Le

1,145
1
12
20

How can I handle Javascript in a Perl web crawler?

4 Answers4

Linked

Related