Downloading html with java not working

Question

So I'm trying to download html page from URL;

public static void getHtml(){
    URL url;
    InputStream is = null;
    BufferedReader br;
    String line;

    try {
        url = new URL(URL);
        is = url.openStream();  
        br = new BufferedReader(new InputStreamReader(is));

        while ((line = br.readLine()) != null) {
            System.out.println(line);
        }

    }catch(Exception e){

    } finally {
        try {
            if (is != null) is.close();
        } catch (IOException ioe) {

        }
    }
}

The thing is, it gives me not the HTML I want, but the following:

<html>
 <head>
  <title>loading</title>
 </head>
 <body>
  <p>Please wait...</p>
       <script>document.cookie="a=3c5hb1488cb3eghv3r456t12234jfyr7g;path=/;";location.href=document.location.pathname;</script>
 </body>
</html>

How do I download the content of the webpage directly? I also tried jsoup but it gave the same result. Also tried Apache - same.

I know this isn't an answer, but do you HAVE to download a webpage using java? there are easier ways to do it than this — Domecraft, Nov 19 '13 at 02:21
try a different web page first to confirm it is not page specific — ghostbust555, Nov 19 '13 at 02:23
@Domecraft I need to parse html to get a js func there, so yeah, I have to download it — Valera, Nov 19 '13 at 02:25
@ghostbust555 I'm sure it's page specific. It workod OK few days ago, I guess they changed smth there, but I'm still able to look at the source code with my browser. Is there a way to do it programatically? — Valera, Nov 19 '13 at 02:27

score 1 · Accepted Answer · edited May 23 '17 at 11:50

1

Here my guess about the website.

It return this page to first-time visitor
The browser set a cookie, and refresh (redirected to the same url)
With the cookie, the server respond true content

So it works with browser but not java.

You may parse the set cookie script and replay it. "a=3c5hb1488cb3eghv3r456t12234jfyr7g;path=/;"

Refer to the following post for setting cookies on url connect URLConnection with Cookies?

Or use Apache HTTP Client http://hc.apache.org/httpclient-3.x/

edited May 23 '17 at 11:50

Community

1
1

answered Nov 19 '13 at 02:50

user2913985

56
3

Yes, perfect guess, they require cookie and then redirect to main page. Thanks! – Valera Nov 19 '13 at 03:33

Downloading html with java not working

1 Answers1