How to scrape Google SERPs with Jsoup?

Question

I was trying to scrape links from google using 600 different searches, In the process of this I started getting the following error.

Error

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=503, URL=http://ipv4.google.com/sorry/IndexRedirect?continue=http://google.com/search/...

Now I've done my research and it happens because of google scholar ban restricting you to limited searches and need to solve captch to proceed, which jsoup can't do.

Code

Document doc = Jsoup.connect("http://google.com/search?q=" + keyWord)
.userAgent("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
.timeout(5000)
.get();

Answers on the internet are extremely vague and doesn't provide a clear solution, someone did mention cookies can solve this issue but haven't said a single thing about "how" to do it.

till my knowledge error `5**` comes as internal sever problem , did you check your server configuration??(just asking) — bananas, Aug 02 '16 at 10:31
@Radioactive Coffe Can you update ur solution here ?Also ,I got the temporary ban from google .How to resolve it ? — Vito, Jan 23 '17 at 10:33

score 2 · Accepted Answer · edited May 23 '17 at 12:16

Some hints to improve your scraping:

1. Use proxies

Proxies permit you to reduce chances to get caught by a captcha. You should use between 50 and 150 proxies depending on your average result set. Here are two websites that can provide some proxies: SEO-proxies.com or Proxify Switch Proxy.

// Setup proxy
String proxyAdress = "1.2.3.4";
int proxyPort = 1234;
Proxy proxy = new Proxy(Proxy.Type.HTTP, InetSocketAddress.createUnresolved(proxyAdress, proxyPort))

// Fetch url with proxy
Document doc = Jsoup //
               .proxy(proxy) //
               .userAgent("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2") //
               .header("Content-Language", "en-US") //
               .connect(searchUrl) //
               .get();

2. Captchas

If by any mean, you get caught by captcha, you can use some online captcha solving services (Bypass Captcha, DeathByCaptcha to name a few). Below is a generic step by step procedure to get the captcha solved automatically:

Detect captcha error page

--

try {

  // Perform search here...

} catch(HttpStatusException e) {
    switch(e.getStatusCode()) {
        case java.net.HttpURLConnection.HTTP_UNAVAILABLE:
            if (e.getUrl().contains("http://ipv4.google.com/sorry/IndexRedirect?continue=http://google.com/search/...")) {
               // Ask online captcha service for help...
            } else {
               // ...
            }
        break;

        default:
        // ...
    } 
}

Download the captcha image (CI)

--

Jsoup                     //
//.cookie(..., ...)       // Some cookies may be needed...
.connect(imageCaptchaUrl) //
.ignoreContentType(true)  // Needed for fetching image
.execute()                //
.bodyAsBytes();           // byte[] array returned...

Send CI to online captcha service online

--

This part depends on the captcha service API. You can find some services in this 8 best captcha solving services article.

Wait for response... (1-2 second(s) is perfect)
Fill the form with response and send it with Jsoup

The Jsoup FormElement is a life saver here. See this working sample code for details.

3. Some other hints

The Hints for Google scrapers article can give you some more pointers for improving your code. You'll find the first two hints presented here plus some more:

Cookies: clear them on each IP change or don't use them at all
Threads: You should not open two many connections. Firefox limits itself to 4 connections per proxy.
Returned results: append &num=100 to your url to sent less requests
Request rates: Make your requests look human. You should not send more than 500 requests per 24h per IP.

References :

You sir are a no.1 legend, I haven't seen this accurate answer in my entire life, you just made my day and probably gonna help alot of people in future who run across this problem — Radioactive Coffe, Aug 02 '16 at 10:50

score 1 · Answer 2 · answered Apr 06 '18 at 21:35

As an alternative to Stephan answer, you can use this package to get the Google search results without the hassle of proxies. Code sample:

Map<String, String> parameter = new HashMap<>();
parameter.put("q", "Coffee");
parameter.put("location", "Portland");
GoogleSearchResults serp = new GoogleSearchResults(parameter);

JsonObject data = serp.getJson();
JsonArray results = (JsonArray) data.get("organic_results");
JsonObject first_result = results.get(0).getAsJsonObject();
System.out.println("first coffee: " + first_result.get("title").getAsString());

Project Github

This project use "Serpapi" that is a paid product. – Genaut May 09 '21 at 16:08 — Genaut, May 09 '21 at 16:08

How to scrape Google SERPs with Jsoup?

2 Answers2

1. Use proxies

2. Captchas

3. Some other hints

References :