403 error while getting the google result using jsoup

Question

I'm trying to get Google results using the following code:

Document doc = con.connect("http://www.google.com/search?q=lakshman").timeout(5000).get();

But I get this exception:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403,URL=http://www.google.com/search?q=lakshman

A 403 error means the server is forbidding access, but I can load this URL in a web browser just fine. Why does Jsoup get a 403 error?

It's probably the absence of a `USER_AGENT` header that triggers the 403. I think this is against Google's TOS in any case — Pekka, Jan 22 '13 at 20:32
oh.thanks for the warning.then is there a way to get the google result by automating? — lakshman, Jan 22 '13 at 20:41
I think they used to have a search API, but I'm not sure what the status is — Pekka, Jan 22 '13 at 20:41
You can set user-agent using jsoup: http://stackoverflow.com/questions/6581655/jsoup-useragent-how-to-set-it-right — Aravind Yarram, Jan 22 '13 at 20:44
http://stackoverflow.com/questions/10120849/jsoup-connect-throws-403-error-while-apache-httpclient-is-able-to-fetch-the-cont/10136686#10136686 — user1498298, Feb 02 '14 at 17:22
@lakshman can u demonstrate your solution .I got this problem too. — Vito, Jan 24 '17 at 09:53
@Vito: Solution is to add the user agent property as mentioned by Liang or use the search API. — lakshman, Jan 24 '17 at 14:09
.@lakshman I have added this`userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36")` or modified to my own browser version .The 403 error still exists . — Vito, Jan 24 '17 at 16:23
@Vito userAgent("Mozilla") worked and below 2 options didn't work //final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"; //final String USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"; — vikramvi, Sep 02 '22 at 06:03

score 38 · Accepted Answer · answered Mar 18 '14 at 02:44

38

You just need to add the UserAgent property to HTTP header as follows:

Jsoup.connect(itemUrl)
     .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36")
     .get()

answered Mar 18 '14 at 02:44

Liang

767
10
13

Thanks! Works great! – ricardogobbo May 28 '16 at 04:26
userAgent("Mozilla") worked and below 2 options didn't work //final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"; //final String USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"; – vikramvi Sep 02 '22 at 06:02
Interestingly, in my case the user agent string in the answer didn't resolve the problem but my browser's actual user agent string, `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/116.0`, did. – joriki Aug 13 '23 at 18:23

score 6 · Answer 2 · answered Dec 16 '13 at 22:01

6

Google doesn't allow robots, you couldn't use jsoup to connect google. You can use the Google Web Search API (Deprecated) but the number of requests you may make per day will be limited.

answered Dec 16 '13 at 22:01

Rowandish

2,655
3
31
52

Phani Rahul · Answer 3 · 2014-06-03T17:33:46.447

3

Actually, you can evade 403 error by just adding a user-agent

doc = Jsoup.connect(url).timeout(timeout)
                    .userAgent("Mozilla")

But that is against the google policy I think.

EDIT: Google catches robots quicker than you think. You can however, use this as a temporary solution.

edited Jun 03 '14 at 17:33

answered Feb 10 '14 at 22:10

Phani Rahul

840
7
22

Java Enthusiast · Answer 4 · 2014-04-29T20:05:34.067

1

Replace statement

Document doc =con.connect("http://www.google.com/search?q=lakshman").timeout(5000).get();

with statement

Document doc=Jsoup.connect("http://www.google.com/search?q=lakshman").userAgent("Chrome").get();

edited Apr 29 '14 at 20:05

answered Apr 29 '14 at 20:00

Java Enthusiast

654
7
19

score 1 · Answer 5 · answered Aug 23 '16 at 23:11

1

try this:

Document doc =con.connect("http://www.google.com/search?q=lakshman").ignoreHttpErrors(true).timeout(5000).get();

in case userAgent did not work Just like it didn't for me.

answered Aug 23 '16 at 23:11

Mahdi-Malv

16,677
10
70
117

score 1 · Answer 6 · answered Jan 29 '18 at 21:13

In some cases you need to set a referrer. It helped in my case.

The full source here

    try{

        String strText = 
                Jsoup
                .connect("http://www.whatismyreferer.com")
                .referrer("http://www.google.com")
                .get()
                .text();

        System.out.println(strText);

    }catch(IOException ioe){
        System.out.println("Exception: " + ioe);
    }

403 error while getting the google result using jsoup

6 Answers6

Linked