(Java) Get the first n results of google as links

Question

First of all, I looked for similar questions, but I couldn't find the answer I need. So, pardon me if this question is not unique and new.

I want to get the first N (probably 5 or 10) results of google as links. At the moment I have something like this:

String url="http://www.google.com/search?q=";
String charset="UTF-8";
String key="java";
String query = String.format("%s",URLEncoder.encode(key, charset));
URLConnection con = new URL(url+ query).openConnection();
//next line is to trick Google who is blocking the default UserAgent
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
    System.out.println(inputLine);
in.close();

This gives me the complete google html code of this searching, but I only want to get the raw links of the first n results. How do I manage that?

Thanks in advance.

Try searching the html for "href" which should denote a link following. — Winston Yang, Jul 09 '18 at 20:31
There are some good parsing solutions, but you may also want to read the comment thread on [this SO answer](https://stackoverflow.com/a/4082976/17300) discussing the deprecated search API and violations of Google's terms of service. — Stephen P, Jul 10 '18 at 00:42

Pieter Mantel · Answer 1 · 2018-07-10T06:08:31.390

I've done some html investigating, and you have to search in the string for:

<h3 class="r"><a href="/url?q=

After that, a link follows, which continues to a double quote. I'll make a script soon.
EDIT
This should get the first n links when searching in google the string key:

public static String[] getLinks(String key, int n) throws MalformedURLException, IOException {
    String url = "http://www.google.com/search?q=";
    String charset = "UTF-8";
    String query = String.format("%s", URLEncoder.encode(key, charset));
    URLConnection con = new URL(url + query).openConnection();
    con.setRequestProperty("User-Agent",
            "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
    BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
    String inputLine;
    String wholeThing = "";
    while ((inputLine = in.readLine()) != null) wholeThing += inputLine;
    in.close();

    List<String> strings = new ArrayList<String>();
    String search = "<h3 class=\"r\"><a href=\"/url?q=";
    int stringsFound = 0;
    int searchChar = search.length();
    while(stringsFound < n && searchChar <= wholeThing.length()) {
        if(wholeThing.substring(searchChar - search.length(), searchChar).equals(search)) {
            int endSearch = 0;
            while(!wholeThing.substring(searchChar + endSearch, searchChar + endSearch + 4).equals("&amp")) {
                endSearch++;
            }
            strings.add(wholeThing.substring(searchChar, searchChar + endSearch));
            stringsFound++;
        }
        searchChar++;
    }
    String[] out = new String[strings.size()];
    for(int i = 0; i < strings.size(); i++) {
        out[i] = strings.get(i);
    }
    return out;
}

Make sure to import java.util.list, not java.awt.list!

sprinter · Accepted Answer · 2018-07-10T21:29:48.493

You might want to try the jsoup library as it takes a lot of the effort out of parsing web pages:

Elements links = Jsoup.connect("https://www.google.com.au/search?q=fred")
    .get().select("h3.r").select("a");
for (Element link : links)
    System.out.println(link);

Elements extends ArrayList<Element> so you can access the first n elements with:

for (int i = 0; i < n; i++)
    System.out.println(links.get(i));

Or, using streams:

links.stream().limit(n)...

If you want the raw url only:

link.attr("href")

So putting all that together, the following will print the first 5 raw links for a google search for the term "fred":

Jsoup.connect("https://www.google.com.au/search?q=fred").get()
    .select("h3.r").select("a")
    .stream()
    .limit(5)
    .map(l -> l.attr("href"))
    .forEach(System.out::println);

(Java) Get the first n results of google as links

2 Answers2