Get the redirected URL of a very specific URL (in Java)

Question

How can I get the redirected URL of http://at.atwola.com/?adlink/5113/1649059/0/2018/AdId=4041444;BnId=872;itime=15692006;impref=13880156912668385284; in Java?

My code (given below) is constructed according to answers to similar questions on stack-overflow (https://stackoverflow.com/a/5270162/1382251 in particular).

But it just yields the original URL. I suspect that there are other similar cases, so I would like to resolve this one in specific and use the solution in general.

    String ref = "http://at.atwola.com/?adlink/5113/1649059/0/2018/AdId=4041444;BnId=872;itime=15692006;impref=13880156912668385284;";
    try
    {
        URLConnection con1 = new URL(ref).openConnection();
        con1.connect();
        InputStream is = con1.getInputStream();
        URL url = con1.getURL();
        is.close();
        String finalPage = url.toString();
        if (finalPage.equals(ref))
        {
            HttpURLConnection con2 = (HttpURLConnection)con1;
            con2.setInstanceFollowRedirects(false);
            con2.connect();
            if (con2.getResponseCode()/100 == 3)
                finalPage = con2.getHeaderField("Location");
        }
        System.out.println(finalPage);
    }
    catch (Exception error)
    {
        System.out.println("error");
    }

score 1 · Accepted Answer · edited May 23 '17 at 12:28

I played a bit with your URL with telnet, wget, and curl and I noticed that in some cases the server returns response 200 OK, and sometimes 302 Moved Temporarily. The main difference seems to be the request User-agent header. Your code works if you add the following before con1.connect():

con1.setRequestProperty("User-Agent","");

That is, with empty User-Agent (or if the header is not present at all), the server issues a redirect. With the Java User-Agent (in my case User-Agent: Java/1.7.0_45) and with the default curl User-Agent (User-Agent: curl/7.32.0) the server responds with 200 OK.

In some cases you might need to also set:

System.setProperty("http.agent", "");

See Setting user agent of a java URLConnection

The server running the site is the Adtech Adserver and apparently it is doing user agent sniffing. There is a long history of user agent sniffing. So it seems that the safest thing to do would be to set the user agent to Mozilla:

con1.setRequestProperty("User-Agent","Mozilla"); //works with your code for your URL

Maybe the safest option would be to use a user agent used by some of the popular web browsers.

Get the redirected URL of a very specific URL (in Java)

1 Answers1