HTTP URL connection response

Question

I am trying to hit the URL and get the response from my Java code.

I am using URLConnection to get this response. And writing this response in html file.

When opening this html in browser after executing the java class, I am getting only google home page and not with the results.

Whats wrong with my code, my code here,

    FileWriter fWriter = null;
    BufferedWriter writer = null;

    URL url = new URL("https://www.google.co.in/?gfe_rd=cr&ei=aS-BVpPGDOiK8Qea4aKIAw&gws_rd=ssl#q=google+post+request+from+java");
    byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
    String encoding = new String(encodedBytes);
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setRequestMethod("GET");
    connection.setRequestProperty("User-Agent", "Mozilla/5.0");
    connection.setRequestProperty("Accept-Charset", "UTF-8");
    connection.setDoInput(true);        
    connection.setRequestProperty("Authorization", "Basic " + encoding);
    connection.connect();

    InputStream content = (InputStream) connection.getInputStream();
    BufferedReader in = new BufferedReader(new InputStreamReader(content));
    String line;

    try {
        fWriter = new FileWriter(new File("f:\\fileName.html"));
        writer = new BufferedWriter(fWriter);
        while ((line = in.readLine()) != null) {
            String s = line.toString();
            writer.write(s);
            }
    writer.close();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Same code works couple of days back, but not now.

score 1 · Answer 1 · edited May 23 '17 at 12:23

1

This method of searching is not advised is supposed to fail, you must use google search APIs for this kind of work.

Note: Google uses some redirection and uses token, so even if you will find a clever way to handle it, it is ought to fail in long run.

Edit:

This is a sample of how using Google search APIs you can get your work done in reliable way; please do refer to the source for more information.

public static void main(String[] args) throws Exception {
    String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
    String search = "stackoverflow";
    String charset = "UTF-8";

    URL url = new URL(google + URLEncoder.encode(search, charset));
    Reader reader = new InputStreamReader(url.openStream(), charset);
    GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);

    // Show title and URL of 1st result.
    System.out.println(results.getResponseData().getResults().get(0).getTitle());
    System.out.println(results.getResponseData().getResults().get(0).getUrl());
}

edited May 23 '17 at 12:23

Community

1
1

answered Dec 29 '15 at 07:22

anand

1,506
14
28

This returns the resulting page as response two days back, but not now. – Dec 29 '15 at 07:25
@user4120936 please see the **Note** I added for explanation on this. – anand Dec 29 '15 at 07:28
can you suggest some url for get this working please? using google api – Dec 29 '15 at 07:29
@user4120936 please take a look on the edit and read the complete information from the link given in source. – anand Dec 29 '15 at 07:36

score 1 · Answer 2 · edited May 23 '17 at 11:52

1

The reason is that this url does not return search results it self. You have to understand google's working process to understand it. Open this url in your browser and view its source. You will only see lots of javascript there.

Actually, in a short summary, google uses Ajax requests to process search queries.

To perform required task you either have to use a headless browser (the hard way) which can execute javascript/ajax OR better use google search api as directed by anand.

edited May 23 '17 at 11:52

Community

1
1

answered Dec 29 '15 at 07:25

Tariq

2,489
11
36
61

This returns the resulting page as response two days back, but not now. – Dec 29 '15 at 07:27
@user4120936 same url was used? – Tariq Dec 29 '15 at 07:32

HTTP URL connection response

2 Answers2