85

I'm trying to parse a webpage using Java with URLConnection. I try to set up the user-agent like this:

java.net.URLConnection c = url.openConnection();
c.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");

But the resulting user agent is the one I specify, with "Java/1.5.0_19" appended to the end. Is there a way to truly set the user agent without this addition?

DiglettPotato
  • 1,654
  • 3
  • 15
  • 19

4 Answers4

95

Just for clarification: setRequestProperty("User-Agent", "Mozilla ...") now works just fine and doesn't append java/xx at the end! At least with Java 1.6.30 and newer.

I listened on my machine with netcat(a port listener):

$ nc -l -p 8080

It simply listens on the port, so you see anything which gets requested, like raw http-headers.

And got the following http-headers without setRequestProperty:

GET /foobar HTTP/1.1
User-Agent: Java/1.6.0_30
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

And WITH setRequestProperty:

GET /foobar HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

As you can see the user agent was properly set.

Full example:

import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;


public class TestUrlOpener {

    public static void main(String[] args) throws IOException {
        URL url = new URL("http://localhost:8080/foobar");
        URLConnection hc = url.openConnection();
        hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");

        System.out.println(hc.getContentType());
    }

}
juwens
  • 3,729
  • 4
  • 31
  • 39
74

Off hand, setting the http.agent system property to "" might do the trick (I don't have the code in front of me).

You might get away with:

 System.setProperty("http.agent", "");

but that might require a race between you and initialisation of the URL protocol handler, if it caches the value at startup (actually, I don't think it does).

The property can also be set through JNLP files (available to applets from 6u10) and on the command line:

-Dhttp.agent=

Or for wrapper commands:

-J-Dhttp.agent=
Tom Hawtin - tackline
  • 145,806
  • 30
  • 211
  • 305
  • How would I do that? c.setRequestProperty("http.agent","");? I'm assuming somewhere else... – DiglettPotato Mar 27 '10 at 16:18
  • 1
    yeah... that work like a charm! just do a: System.setProperty("http.agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2"); And you're ready to go!! :) – eduardo.lopes Jul 22 '16 at 12:39
5

its work for me set the User-Agent in the addRequestProperty.

URL url = new URL(<URL>);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.addRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0");
Bachan Joseph
  • 347
  • 4
  • 5
  • No need to cast to HttpURLConnection. – jechterhoff May 16 '18 at 10:11
  • 1
    @jechterhoff it's required in Java 8 – FonzTech Jun 26 '18 at 15:37
  • @FonzTech I don't see why the cast would be required in Java 8. The following works in my program (compliance is Java 1.8):URL clSourceUrl = new URL(clSource); URLConnection urlConn = clSourceUrl.openConnection(); urlConn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"); Can you elaborate, please? – jechterhoff Jun 29 '18 at 14:25
  • @jechterhoff the user who gave the answer used `HttpURLConnection`. You would have been right if he used `URLConnection`. Anyway, if you try to compile this line `HttpURLConnection c = new URL("http://www.google.com").openConnection();` for example, `javac` will give you `error: incompatible types` on that line. I'm using Java 1.8.0 r172, so the most recent release of Java 8 – FonzTech Jul 02 '18 at 13:54
  • @FonzTech Ah, now I see what you mean. I should have been more precise in my first comment (sorry about that): What I was trying to say is that you don't need to use HttpURLConnection at all here. Apparently you can also just use a URLConnection, as shown in my previous comment. I thought that this could improve the answer a tiny bit. In any case, you are right that `HttpURLConnection c = new URL("http://www.google.com").openConnection();` certainly requires a cast to HttpURLConnection - in Java 8 and also previous versions of Java. – jechterhoff Jul 03 '18 at 14:57
  • @jechterhoff Don't worry. You are right about `HttpURLConnection` being overkill here, but since the user who gave the answer used it, the casting to the derived class was necessary. – FonzTech Jul 04 '18 at 09:28
2

HTTP Servers tend to reject old browsers and systems.

The page Tech Blog (wh): Most Common User Agents reflects the user-agent property of your current browser in section "Your user agent is:", which can be applied to set the request property "User-Agent" of a java.net.URLConnection or the system property "http.agent".

Sam Ginrich
  • 661
  • 6
  • 7