Java getting source code from a website

Question

I have a problem once again where I cant find the source code because its hidden or something... When my java program indexes the page it finds everything but the info i need... I assume its hidden for a reason but is there anyway around this?

Its just a bunch of tr/td tags that show up in firebug but dont show up when viewing the page source or when i do below

URL url = new URL("my url");
            URLConnection yc = url.openConnection();
            BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
            String inputLine;
            while ((inputLine = in.readLine()) != null) {

I really have no idea how to attempt to get the info that i need...

score 2 · Accepted Answer · answered Jul 04 '11 at 06:06

The reason for this behavior is because probably those tags are dynamically injected into the DOM using javascript and are not part of the initial HTML which is what you can fetch with an URLConnection. They might even be created using AJAX. You will need a javascript interpreter on your server if you want to fetch those.

score 0 · Answer 2 · answered Jul 04 '11 at 06:07

0

If they don't show up in the page source, they're likely being added dynamically by Javascript code. There's no way to get them from your server-side script short of including a javascript interpreter, which is rather high-overhead.

The information in the tags is presumably coming from somewhere, though. Why not track that down and grab it straight from there?

answered Jul 04 '11 at 06:07

Nick Johnson

100,655
16
128
198

Interesting... How would I go about hunting that down? – Jon Storm Jul 04 '11 at 06:19
@Jon, install FireBug for Firefox, activate it and load the page; then switch to FireBugs's network mode to see all traffic generated, along with JavaScript-generated events. – Tassos Bassoukos Jul 04 '11 at 06:55

score 0 · Answer 3 · answered Jul 04 '11 at 06:16

0

Try Using Jsoup.

Document doc = doc=Jsoup.parse("http:\\",10000);
System.out.print(doc.toString());

answered Jul 04 '11 at 06:16

Rasel

15,499
6
40
50

score 0 · Answer 4 · edited May 23 '17 at 12:19

0

Assuming that the issue is that the "missing" content is being injected using javascript, the following SO Question is pertinent:

What's a good tool to screen-scrape with Javascript support?

edited May 23 '17 at 12:19

Community

1
1

answered Jul 04 '11 at 06:29

Stephen C

698,415
94
811
1,216

Java getting source code from a website

4 Answers4