How can a java program automatically follow redirection in 303 http status code response

Question

I tried to access this url in my java program but I got this strange message instead of the page content as I was expecting.

How can I avoid this?

<!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
 <head> 
  <title>303 See Other</title> 
 </head>
 <body> 
  <h1>See Other</h1> 
  <p>The answer to your request is located <a href="https://www.wikidata.org/wiki/Special:EntityData/P26">here</a>.</p>  
 </body>
</html>

In a browser though I can navigate there easily. Is there some function or library I can use to evoke that functionality from my java program?

for (String url : list_of_relation_URLs) 
{
    //System.out.println( url );

    //go to relation url
    String URL_czech = url;

    System.out.println( url  );

    URL wikidata_page = new URL(URL_czech);
    HttpURLConnection wiki_connection = (HttpURLConnection)wikidata_page.openConnection();
    InputStream wikiInputStream = null;

    try 
    {
        // try to connect and use the input stream
        wiki_connection.connect();
        wikiInputStream = wiki_connection.getInputStream();
    } 
    catch(IOException error) 
    {
        // failed, try using the error stream
        wikiInputStream = wiki_connection.getErrorStream();
    }

    // parse the input stream using Jsoup
    Document docx = Jsoup.parse(wikiInputStream, null, wikidata_page.getProtocol()+"://"+wikidata_page.getHost()+"/");

    System.out.println( docx.toString() );  
}

I'm trying to do basically the opposite of what is going on here.

is there a way to automatically follow it as a browser does? — , Apr 28 '15 at 13:50
see also http://stackoverflow.com/questions/2659000/java-how-to-find-the-redirected-url-of-a-url — Tilman Hausherr, Apr 28 '15 at 13:54
i tried with `wiki_connection.setInstanceFollowRedirects(true);` but still it didn't work. — , Apr 28 '15 at 13:56
@TilmanHausherr they're doing the opposite of what I want to do isn't it? — , Apr 28 '15 at 13:59
@Yamada_Tarō yes, but the URLs should still give you some insight. — Tilman Hausherr, Apr 28 '15 at 14:00

Fenton · Accepted Answer · 2015-04-28T13:54:48.520

1

When you receive a 303 status code, you simply need to make a second request to the URL supplied with the 303.

The new URL is stored in the Location header.

In your case, you will need to keep following until you get a different status code as you will be redirected two times.

303: Location:"https://www.wikidata.org/wiki/Special:EntityData/P26"

303: Location:"https://www.wikidata.org/wiki/Property:P26"

And yes... if you are using a HttpURLConnection you can ask it to do this for you.

conn.setInstanceFollowRedirects(true);

edited Apr 28 '15 at 13:54

answered Apr 28 '15 at 13:52

Fenton

241,084
71
387
401

what about this `setInstanceFollowRedirects(true);` – Apr 28 '15 at 13:53
i tried `wiki_connection.setInstanceFollowRedirects(true);` is that what you mean? – Apr 28 '15 at 13:56
Yes - your question didn't have code when I answered, so I assumed a HttpUrlConnection called `conn`. – Fenton Apr 28 '15 at 14:00
but actually I tried that and still it didn't work :/ same. no change in the response. – Apr 28 '15 at 14:01
It will follow redirects on the same protocol. If you switch between http/https it won't follow (for security reasons). – Fenton Apr 28 '15 at 14:46

score 0 · Answer 2 · answered Apr 28 '15 at 14:13

this is the perfect answer

try {

String url = "http://www.twitter.com";

URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
conn.setReadTimeout(5000);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");

System.out.println("Request URL ... " + url);

boolean redirect = false;

// normally, 3xx is redirect
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
    if (status == HttpURLConnection.HTTP_MOVED_TEMP
        || status == HttpURLConnection.HTTP_MOVED_PERM
            || status == HttpURLConnection.HTTP_SEE_OTHER)
    redirect = true;
}

System.out.println("Response Code ... " + status);

if (redirect) {

    // get redirect url from "location" header field
    String newUrl = conn.getHeaderField("Location");

    // get the cookie if need, for login
    String cookies = conn.getHeaderField("Set-Cookie");

    // open the new connnection again
    conn = (HttpURLConnection) new URL(newUrl).openConnection();
    conn.setRequestProperty("Cookie", cookies);
    conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
    conn.addRequestProperty("User-Agent", "Mozilla");
    conn.addRequestProperty("Referer", "google.com");

    System.out.println("Redirect to URL : " + newUrl);

}

BufferedReader in = new BufferedReader(
                          new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer html = new StringBuffer();

while ((inputLine = in.readLine()) != null) {
    html.append(inputLine);
}
in.close();

System.out.println("URL Content... \n" + html.toString());
System.out.println("Done");

How can a java program automatically follow redirection in 303 http status code response

2 Answers2