Check for broken links

Question

I am trying to find all the broken links in the webpage using Java. Here is the code:

   private static boolean isLive(String link){

    HttpURLConnection urlconn = null;
    int res = -1;
    String msg = null;
    try{

        URL url = new URL(link);
        urlconn = (HttpURLConnection)url.openConnection();
        urlconn.setConnectTimeout(10000);
        urlconn.setRequestMethod("GET");
        urlconn.connect();
        String redirlink = urlconn.getHeaderField("Location");
        System.out.println(urlconn.getHeaderFields());
        if(redirlink != null && !url.toExternalForm().equals(redirlink))
            return isLive(redirlink);
        else
            return urlconn.getResponseCode()==HttpURLConnection.HTTP_OK;

    }catch(Exception e){

      System.out.println(e.getMessage());
      return false;

    }finally{

        if(urlconn != null)
            urlconn.disconnect();

    }


}

public static void main(String[] s){

    String link = "http://www.somefakesite.net";
    System.out.println(isLive(link));

}

Code referred from http://nscraps.com/Java/146-program-code-broken-link-checker.htm.

This code gives HTTP 200 status for all webpages including the broken ones. For example http://www.somefakesite.net/ gives the following header fields:

{null=[HTTP/1.1 200 OK], Date=[Sun, 15 May 2011 18:51:29 GMT], Transfer-Encoding=[chunked], Keep-Alive=[timeout=4, max=100], Connection=[Keep-Alive], Content-Type=[text/html], Server=[Apache/2.2.15 (Win32) PHP/5.2.12], X-Powered-By=[PHP/5.2.9-1]}

Even though such sites do not exist, how to classify it as a broken link?

are you 100% sure of what sites you are testing? try adding a `System.out.println(link);` at the top of that method. — Mat, May 15 '11 at 19:03
@Mat: Yes, i did add the print statement and it is testing the correct link which is passed from main — user754740, May 16 '11 at 06:28

SJuan76 · Accepted Answer · 2011-05-17T09:54:41.467

Maybe the issue is that currently lots of webserver and DNS providers detect those "broken" links and redirect you to their "not found" pages.

Test it against an URL that you know sends the 404 code (it shows the browser original message).

EDIT to answer the comment by the author (as it is too long to fit in a comment): I do not see an easy answer for your problem, but there are several different types of failures:

For DNS failures that are redirected (an URL that cannot be found by the DNS, and you get redirected to another page). All redirections (if you are redirected) will likely go to the same page (provided by your ISP/DNS provider), you can check for that. Of course, if you try with another ISP/DNS provider the page might be different. If you are not being redirected then you will get a connection error.
For a server with valid DNSs but not working (for example, google.com goes down), there should be a connection error.
For a resource ("page") missing in a server, it is more difficult. 404 means it is broken, but if the server does not send it there is little more to do. A redirection might be useful to flag a link as dubious, but it should be manually checked later because it is not only used for capturing missing links (for example, www.google.com redirects me www.google.es)

Also, if this was not the case, "somefakesite.com" would return a connection error, not a 404 status (because a missing resource in a server that is working is not the same that a missing server). — SJuan76, May 15 '11 at 19:10
I tested with 2 broken links and both of them displayed different status. 1. http://legacy.ncsu.edu/ECE566 {null=[HTTP/1.1 404 Not Found] status - 404 2. http://somefakewebsite.net {null=[HTTP/1.1 200 OK] status - 200 Now the question is should i have to check the domain header for redirected broken links? Appreciate your help. — user754740, May 16 '11 at 19:35
@user754740 I tried to answer with an edit to my post; unfortunately I know of no easy solution. — SJuan76, May 17 '11 at 09:55

Check for broken links

1 Answers1