google search and print result in java

Question

Here I expect to perform a search on Google and get the result as separate output file by using the terminal. In the terminal, only the keyword to be searched will be entered. This is what I did.

import java.io.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Scanner;


public class GOOGLE {

    public static void main(String[] args) {

        Scanner scan = new Scanner(System.in);
        System.out.println("Enter search key");
        String key = scan.nextLine();
        String keyUpdated = key.replaceAll("\\s", "+");

        System.out.println(keyUpdated);

        try {

            URL url = new URL("https://www.google.lk/search?q=" + keyUpdated);

            BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
            BufferedWriter writer = new BufferedWriter(new FileWriter(key + ".html"));

            String line;

            while ((line = reader.readLine()) != null) {

                writer.write(line);

                writer.newLine();
            }


            reader.close();
            writer.close();
        } catch (MalformedURLException e) {

            e.printStackTrace();

        } catch (IOException e) {

            e.printStackTrace();

        }

    }

}

and this is what i got when i entered "computer science" as key word.

java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.google.lk/search?q=computer+science
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at java.net.URL.openStream(URL.java:1045)
    at GOOGLE.main(Google.java:23)

Possible duplicate of [Why does Google Search return HTTP Error 403?](https://stackoverflow.com/questions/600536/why-does-google-search-return-http-error-403) — user2340612, Jan 27 '18 at 18:45

score 1 · Answer 1 · answered Jan 27 '18 at 18:53

1

The browser signature has caused the request to be rejected.

Have a look at this thread:

Why do I get a 403 error when I try open a URL

answered Jan 27 '18 at 18:53

Eric Snider

11
2

score 0 · Accepted Answer · answered Jan 27 '18 at 19:41

As explained in the answer linked from my comment or in @Eric Snider's answer, the issue is with the user agent used to call Google. In order to provide a valid user agent, you can add a the following option to the VM (or use another user agent, if you prefer):

java -Dhttp.agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0" <program_name>

Or at runtime, using the following method:

System.setProperty("http.agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0");

google search and print result in java

2 Answers2