403 error in accessing an URL but works fine in browsers

Question

String url = "http://maps.googleapis.com/maps/api/directions/xml?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false";

        URL google = new URL(url);
        HttpURLConnection con = (HttpURLConnection) google.openConnection();

and I use BufferedReader to print the content I get 403 error

The same URL works fine in the browser. Could any one suggest.

score 12 · Answer 1 · answered Aug 29 '14 at 05:33

The reason it works in a browser but not in java code is that the browser adds some HTTP headers which you lack in your Java code, and the server requires those headers. I've been in the same situation - and the URL worked both in Chrome and the Chrome plugin "Simple REST Client", yet didn't work in Java. Adding this line before the getInputStream() solved the problem:

                connection.addRequestProperty("User-Agent", "Mozilla/4.0");

..even though I have never used Mozilla. Your situation might require a different header. It might be related to cookies ... I was getting text in the error stream advising me to enable cookies.

Note that you might get more information by looking at the error text. Here's my code:

        try {
            HttpURLConnection connection = ((HttpURLConnection)url.openConnection());
            connection.addRequestProperty("User-Agent", "Mozilla/4.0");
            InputStream input;
            if (connection.getResponseCode() == 200)  // this must be called before 'getErrorStream()' works
                input = connection.getInputStream();
            else input = connection.getErrorStream();
            BufferedReader reader = new BufferedReader(new InputStreamReader(input));
            String msg;
            while ((msg =reader.readLine()) != null)
                System.out.println(msg);
        } catch (IOException e) {
            System.err.println(e);
        }

score 3 · Answer 2 · answered Sep 20 '11 at 13:15

3

HTTP 403 is a Forbidden status code. You would have to read the HttpURLConnection.getErrorStream() to see the response from the server (which can tell you why you have been given a HTTP 403), if any.

answered Sep 20 '11 at 13:15

Buhake Sindi

87,898
29
167
228

Here's how to read the error stream: http://stackoverflow.com/a/9129991/10592 You need to call connection.getResponseCode() before getErrorStream() will work. – Tim Cooper Aug 29 '14 at 05:14
@Tim Coope, it's funny as you posted the answer that **I** posted. – Buhake Sindi Aug 29 '14 at 08:09

score 2 · Answer 3 · answered Sep 20 '11 at 13:40

2

This code should work fine. If you have been making a number of requests, it is possible that Google is just throttling you. I have seen Google do this before. You can try using a proxy to verify.

answered Sep 20 '11 at 13:40

smp7d

4,947
2
26
48

score 0 · Answer 4 · answered Apr 22 '13 at 20:18

0

I know this is a bit late, but the easiest way to get the contents of a URL is to use the Apache HttpComponents HttpClient project: http://hc.apache.org/httpcomponents-client-ga/index.html

answered Apr 22 '13 at 20:18

Dave F

821
1
9
20

score 0 · Answer 5 · answered Jun 28 '19 at 20:18

you original page (with link) and the targeted linked page are not the same domain.

original-domain and target-domain.

I found the difference is in request header:

with 403 forbidden error,

         request header have one line:
                        Referer: http://original-domain/json2tree/ipfs/ipfsList.html

when I enter url, no 403 forbidden, the request header does NOT have above line referer: original-domain

I finally figure out how to fix this error!!!

on your original-domain web page, you have to add

              <meta name="referrer" content="no-referrer" />

it will remove or prevent sending the Referer in header, works both for links and for Ajax requests made

this is how you remove referer in request header https://stackoverflow.com/questions/6817595/remove-http-referer — hoogw, Jun 28 '19 at 20:32

Mob · Answer 6 · 2011-09-20T13:40:30.613

0

Most browsers automatically encode URLs when you enter them, but the Java URL function doesn't. You should Encode the URL with URLEncoder URL Encoder

edited Sep 20 '11 at 13:40

answered Sep 20 '11 at 13:16

Mob

10,958
6
41
58

403 error in accessing an URL but works fine in browsers

6 Answers6