Not getting correct HTTP Response Code in Selenium WebDriver with java

Question

I am trying to do automation of find all the broken link of a page. I have gone through so many articles here but none helped. The real problem I am facing is I am not able to get(returing) the correct httpresponse code. Below is the code:

public static int getResponseCode(String urlString) {

    try {
        URL url = new URL(urlString);
        HttpURLConnection connection = (HttpURLConnection)url.openConnection();
        connection.setRequestMethod("GET");
        connection.connect();
        return connection.getResponseCode();

    } catch (Exception e) {

    }
    return -1;
}

what is the OUTPUT (HTTP response code) you observed? what you expected & what have you observed? pls share the details. — Naveen Kumar R B, Feb 28 '17 at 13:31
I am getting -1 only even for the valid links. For valid links I expect 2XX. — Vikas Rathore, Feb 28 '17 at 13:37
In **catch block**, print the exception as `return connection.getResponseCode();` is NOT getting executed, I suspect there are exceptions thrown. Share the stack trace with us. — Naveen Kumar R B, Feb 28 '17 at 13:38
give us one `urlString` as an example which is giving `-1` for valid url. we will try it out. what is the IDE you are using? — Naveen Kumar R B, Feb 28 '17 at 13:48
thanks Naveen. I am using eclipse. below is the stacktrace -1https://www.google.co.in/intl/en/about.html?fg=1 sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target — Vikas Rathore, Feb 28 '17 at 13:50

Vijayan Kani · Answer 1 · 2017-02-28T13:51:17.663

0

you cant get the response using this code alone using java. you need the java selenium driver code for this to be implemented.

use the below code to get the right response:

private static int statusCode;
public static void main(String... args) throws IOException{
    WebDriver driver = new FirefoxDriver();
    driver.manage().window().maximize();
    driver.get("https://www.google.com/");
    driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

    List<WebElement> links = driver.findElements(By.tagName("a"));
    for(int i = 0; i < links.size(); i++){
        if(!(links.get(i).getAttribute("href") == null) && !(links.get(i).getAttribute("href").equals(""))){
            if(links.get(i).getAttribute("href").contains("http")){
                statusCode= intgetResponseCode(links.get(i).getAttribute("href").trim());
                if(statusCode == 403){
                    System.out.println("HTTP 403 Forbidden # " + i + " " + links.get(i).getAttribute("href"));
                }
            }
        }   
    }   
}


public static int getResponseCode(String urlString) throws MalformedURLException, IOException{
        URL url = new URL(urlString);
    HttpURLConnection huc = (HttpURLConnection)url.openConnection();
    huc.setRequestMethod("GET");
    huc.connect();
    return huc.getResponseCode();
    }

Else you can do get the response by setting the response method as "HEAD" [if its a simple test].

hope it helps. Cheers!

edited Feb 28 '17 at 13:51

answered Feb 28 '17 at 13:13

Vijayan Kani

368
3
9

Please provide the code of intgetResponseCode(url) function. Main problem is the function is not returning correct response code. What ever you provided is before and after getting the response code. – Vikas Rathore Feb 28 '17 at 13:40
Its the same code that you're using. The code you've used is just fine. Try and lemme know – Vijayan Kani Feb 28 '17 at 13:42
I tired but statusCode is always getting -1 from getResponseCode(), even for valid links – Vikas Rathore Feb 28 '17 at 13:46
The return statement in the final line is the issue. Updated my answer. Pls check – Vijayan Kani Feb 28 '17 at 13:51
That code is already provided in other thread and its not working. I am getting below error unable to find valid certification path to requested target – Vikas Rathore Feb 28 '17 at 13:55
ok its working for me, by specifying the firefox location in the -- WebDriver driver = new FirefoxDriver(<>); line – Vijayan Kani Feb 28 '17 at 14:03

score 0 · Answer 2 · answered Feb 28 '17 at 13:54

0

The following code worked for me with given example URL:

public static int getResponseCode(String urlString) throws IOException {
//      String url = "http://www.google.com/search?q=mkyong";

        String url  = "https://www.google.co.in/intl/en/about.html?fg=1";

        URL obj = new URL(url);
        HttpURLConnection con = (HttpURLConnection) obj.openConnection();

        // optional default is GET
        con.setRequestMethod("GET");

        //add request header
        con.setRequestProperty("User-Agent", "Mozilla/5.0");
        con.connect();

        int responseCode = con.getResponseCode();
        System.out.println("\nSending 'GET' request to URL : " + url);
        System.out.println("Response Code : " + responseCode);

}

Output I got:

Sending 'GET' request to URL : https://www.google.co.in/intl/en/about.html?fg=1
Response Code : 200

answered Feb 28 '17 at 13:54

Naveen Kumar R B

6,248
5
32
65

The output you got is the exact I am expecting but I am getting error sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target – Vikas Rathore Feb 28 '17 at 14:07
add the ssl certificate to the java certificate store and the error should be gone.. or auto accept any certificate :O – metar Feb 28 '17 at 14:10
@metar: I am trying to do the same thing. Could you please explain me the process/steps. – Vikas Rathore Feb 28 '17 at 14:15
How come you got both the response code & exception? If you are getting `200` code, then that means there is NO exception is thrown. Anyways. add `throws IOException` to the method and try., – Naveen Kumar R B Feb 28 '17 at 14:16
@Naveen: I am expecting(wanted) the same output as you are getting but I am getting error sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target – Vikas Rathore Feb 28 '17 at 14:23
I have NOT added any certificate. did you try by throwing the exception? – Naveen Kumar R B Feb 28 '17 at 15:13
@VikasRathore, see [this link](http://stackoverflow.com/a/2893932/1545397) – metar Feb 28 '17 at 16:06

Not getting correct HTTP Response Code in Selenium WebDriver with java

2 Answers2