Reading dynamic web page content in java

Question

I need help reading the contents of a webpage. Currently i am using the following method to read the contents

BufferedReader in = new BufferedReader(new InputStreamReader(page.openStream())); 
String inputLine;
while ((inputLine = in.readLine()) != null)
{Content = Content + inputLine;}

However with this method there is a problem. . some jsp pages have ajax in them which randomly updates a css class of a webpage like so Javascript code just to give an idea:

if (request.readyState === 4 && request.status === 200) 
{
var type = request.getResponseHeader("Content-Type");
$('.update').empty();
$('.update').append(request.responseText); //update the css class
}

So as a result when this page reader is read through my java method as mentioned above i just get

<div class="update"></div>

although on the screen this class has a value. Now however if i save the page first (by clicking save as in Firefox) then the values appended in the CSS class by jquery are also visible. Is there a method or a way on how i could read the values or obtain the values like firefox does by saving the pages.. I want to read the contents of the entire webpage with the Ajax values present in the string.

On one side i read that this is difficult since the JAvascript in rendered and executed by the browser so i wanted to know does firefox have any apis that might help ? Any suggestions would be appreciated.

You're going to have to render the web page, not just read it with a StreamReader. Google search "Web rendering in Java" (without the quotes) to see if you find something you can work with. — Gilbert Le Blanc, Apr 09 '12 at 14:17

score 4 · Accepted Answer · edited Oct 06 '18 at 06:11

4

You may find the following project useful:

HTMLUnit

Here is also a very informative blog post from Data Big Bang.

edited Oct 06 '18 at 06:11

Amin Heydari Alashti

881
1
9
20

answered Apr 09 '12 at 14:18

mellamokb

56,094
12
110
136

Thanks for the great reply looks like HTMLUnit was exactly what i needed , it does sound promising but haven't used it yet. – Rajeshwar Apr 09 '12 at 14:27

daveb · Answer 2 · 2013-10-11T14:56:30.810

2

Also check out PhantomJS. In the same way that Crowbar is a headless Mozilla browser, PhantomJS is a headless WebKit browser - the engine that Safari and Google Chrome use.

edited Oct 11 '13 at 14:56

answered Apr 09 '12 at 16:08

daveb

74,111
6
45
51

PhantomJS seems charming and suits my need – Truong Ha Oct 08 '13 at 13:08

Reading dynamic web page content in java

2 Answers2

Linked