Language for web scraping JAVASCRIPT content

Question

I think topic ask the question, I usually use PHP for parse/ web scraping, but I have really bad time scraping javascript most cases I cant do it

ex: Parse a div that appears when a javascript its executed.

I readed about RUBY, that have a parser library for javascript, so question is w is the languaje for program a web scraping that will effective scrap javascript generated content ?? Its here a library for PHP like the one for ruby for parse javascript content ?

score 0 · Answer 1 · answered Jun 16 '13 at 11:57

0

You should look at PhantomJS and CasperJS (headless browsers).

answered Jun 16 '13 at 11:57

McMeep

688
2
8
13

score 0 · Answer 2 · answered Jul 03 '13 at 07:35

0

In the ruby world the gem for running Phantomjs would be poltergeist

There is another article about some of the options you have in ruby here too (however they are not all js capable)

answered Jul 03 '13 at 07:35

Niels Kristian

8,661
11
59
117

score 0 · Answer 3 · answered Jul 19 '11 at 20:45

0

There are a handful of strategies for this. Depending on your needs, consider pro grammatically instantiating a browser instance that you can hook into and read the page from.

The idea is, let the browser do the work, as the page is made for a browser and not your bot. You can then tap in and scrape away using a browser plugin that feeds data to your primary application running things.

This may be way overkill for what you need though. I'll leave it up to you to decide.

answered Jul 19 '11 at 20:45

Brad

159,648
54
349
530

I readea bout Ruby:mecanicer, that can do it np, but dont know if there are same libs for Php python etc... I didnt found a library for emulate a web browser – Nonyck Jul 19 '11 at 20:57
@Nonyck, you don't want to emulate a browser, you want to attach to a browser. This is a complex task. – Brad Jul 19 '11 at 21:00
And WWW:Mechanize dont do that ?? – Nonyck Jul 19 '11 at 22:07

score 0 · Answer 4 · answered Jul 19 '11 at 21:16

0

You should look at some GUI-less/headless browsers. There is some written for Java. I didn't find one for PHP.

Look at :

answered Jul 19 '11 at 21:16

Maxime Fafard

370
2
8

score 0 · Answer 5 · answered Jul 19 '11 at 22:35

You can try using something like Selenium, which allows you to automate browser tasks.

On the other hand, you can go into details on what happens when the js code is executed. For example, if the js code is requesting something from the server by POSTing some data, you could emulate that in the regular fashion.

Language for web scraping JAVASCRIPT content

5 Answers5

Linked