How to screen scrape an Ajax site in Java?

Question

I wish to screen scrape several Ajax based websites and simulate clicks which refresh part of the webpage, and then read the updated HTML. Is there any Java library which can do this?

possible duplicate of [How do you screen scrape ajax pages?](http://stackoverflow.com/questions/260540/how-do-you-screen-scrape-ajax-pages) — aioobe, Jun 29 '11 at 17:53
I think @Zubair is looking for a java side solution rather than a general screen scraper.... Either way apache's HtmlUnit is the way to go. — Michael J. Lee, Jun 29 '11 at 18:01
Yes, if possible I would like to use a headless server solution, although if it isn't possible then i will have to automate a browser or something — yazz.com, Jun 29 '11 at 18:15
I did see the other screen scraping question mentioned but the tools it linked to were general purpose tools for the most part — yazz.com, Jun 29 '11 at 19:33

score 6 · Accepted Answer · answered Jun 29 '11 at 17:53

6

Use HtmlUnit it's great for this!! It is a headless browser and has the ability to play with clicks, mouse positions and pretty much everything you would want.

answered Jun 29 '11 at 17:53

Michael J. Lee

12,278
3
23
39

score 1 · Answer 2 · edited May 23 '17 at 11:58

1

I think the only way to do this is to embed a browser so that the Javascript is executed and grab the data when the DOM is updated. This related stack overflow question may help.

edited May 23 '17 at 11:58

Community

1
1

answered Jun 29 '11 at 17:53

Jim Rush

4,143
3
25
27

score 0 · Answer 3 · answered May 31 '13 at 01:02

0

These books should help you (although only the first one is intended to Java developers):

answered May 31 '13 at 01:02

sonnuforevis

1,311
6
22
38

How to screen scrape an Ajax site in Java?

3 Answers3