Android: retrieve information from an internet site

Question

My goal is to retrieve the headline from the Wall Street Journal website (http://online.wsj.com/home-page) and put in in my application on the touch of a button. Here is what I have done so far:

public class NewsFeed extends Activity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstasnceState);
        setContentView(R.layout.activity_newsfeed);

        HttpGet httpGet = new HttpGet('http://online.wsj.com/home-page');
        final String headline = String.valueOf(httpGet);

        button1.setOnClickListener (new View.OnClickListener() {
            @Override
            public void onClick (View v) {
                textview1.setText(headline);
            }
       });

    }

}

How can I change my code so that it correctly displays website information?

use thread or asynctask and use a html parser like jsoup to display the content of the tags — Raghunandan, Jul 05 '14 at 18:50
This question is rather broad as it is -- you basically need to download the page and then parse it, but it depends too much on what you're trying to achieve, e.g. what information you want to extract (titles, summaries?), how you want to display it, &c. Please provide more information (for example, on specific problems you're having) and you might get good answers. — matiash, Jul 10 '14 at 03:56

score 3 · Answer 1 · answered Jul 05 '14 at 18:53

First you need to download the HTML. You can do this with a framework called AsyncHttpClient http://loopj.com/android-async-http/

Then you need to parse the HTML Headline, should be something like the content of h1 , therefore you need a HTML Parser like http://jsoup.org/

score 1 · Answer 2 · edited May 23 '17 at 12:11

You cannot get the response string, if you simply pass URI to HttpGet. You can refer the following stackoverflow URL for understading how to use HttpGet & retrieve the response: How do I use the Simple HTTP client in Android?

Also, if you perform any long running operations on main UI thread, it can cause ANR. So, for avoiding that perform this in a separate thread, Executor etc. You can make use of IntentService or AsyncTask or a Service with Thread. Otherwise, one easy approach is to use any third party library like Volley, Facebook Bolt etc.

score 1 · Answer 3 · answered Jul 10 '14 at 08:46

either use http://loopj.com/android-async-http/ like Mapo said or if you feel it's too complicated use https://code.google.com/p/android-query/

I'd go for the first one, since it supports much more, retries etc. Android Query is good if you want images, facebook / twitter integration etc. very easily

score 1 · Answer 4 · answered Jul 14 '14 at 13:21

Here is an simple proof of concept without any foreign libraries:

        new AsyncTask<String, Integer, String>(){

        @Override
        protected String doInBackground(String... params) {
            try {
                // get HTTP Data
                HttpClient client = new DefaultHttpClient();
                HttpResponse response = client.execute(new HttpGet(params[0])); // We only support the first param 
                String html = EntityUtils.toString(response.getEntity());

                Matcher m = Pattern.compile("<title>(.*?)</title>").matcher(html);
                if(m.find()){
                    Log.i("HeadLine", m.group(1));
                    return m.group(1);
                }

            } catch (Exception e) {
                Log.e("e", e.getMessage(),e);
            }

            // nothing found or an exception occurred
            return null;
        }
        @Override
        protected void onPostExecute(String result) {
            // update your view with the value
            super.onPostExecute(result);
        }

    }.execute("http://online.wsj.com/home-page");

You have definitely to work out the details, but I'm sure you could make that by your self.

score 1 · Answer 5 · answered Jul 15 '14 at 03:31

Grab the relevant feed and parse it as follows:

URL url = new URL("http://example.com/feed.rss");
RssFeed feed = RssReader.read(url);

ArrayList<RssItem> rssItems = feed.getRssItems();
for(RssItem rssItem : rssItems) {
    Log.i("RSS Reader", rssItem.getTitle());
}

Instead of parsing the HTML, which is often messy and change-prone, consume the XML, which exists for you to consume.

score 1 · Answer 6 · answered Jul 16 '14 at 15:04

You can use JSOUP

  //Get the HTML
  Document doc = Jsoup.connect("http://online.wsj.com/europe").get();
  //Get the Headlines
  Elements headlines = doc.select("h2.tipTarget");

  for (Element e : headlines) {
     String headline = e.text();

  }

For avoid exceptions, you must execute prior code in other thread. Example:

protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstasnceState);
        setContentView(R.layout.activity_newsfeed);
        new GetHeadLines().execute();
}


private class GetHeadLines extends AsyncTask<Void, Void, Elements> {

    protected Elements doInBackground(Void... params) {
      //Get the HTML
      Document doc = Jsoup.connect("http://online.wsj.com/europe").get();
      //Get the Headlines
      Elements headlines = doc.select("h2.tipTarget");

      return headlines;
 }

   protected void onPostExecute(Elements headlines) {
        for (Element h : elements) {
                String headline = h.text();
                //Do stufs
        }
    }
}

score 0 · Answer 7 · answered Jul 05 '14 at 18:56

0

HTTPGet is just an object, this why you get a "cryptic" message. You need to execute

httpGet.getMethod();

and you get a string. You also should have a look at HTTPGet APi

answered Jul 05 '14 at 18:56

Aruscher

153
1
10

With a httpGet you only make a get request -> you wouldn't get the page source code. You need to replace your code with HttpGet getRequest = new HttpGet('http://online.wsj.com/home-page'); String getInfos = getRequest.getMethod(); – Aruscher Jul 05 '14 at 19:05
Check again. `getMethod()` returns a fixed string. It does not actually _execute_ the network request! – matiash Jul 05 '14 at 20:08

score 0 · Answer 8 · edited May 23 '17 at 11:49

0

You did not execute your HttpGet!

        HttpClient httpclient = new DefaultHttpClient();
        HttpGet method = new HttpGet("http://online.wsj.com/home-page");
        HttpResponse response = httpclient.execute(method);

Try this link also - HttpClient.execute(HttpPost) on Android 4.2 error

edited May 23 '17 at 11:49

Community

1
1

answered Jul 05 '14 at 19:25

Sergey Shustikov

15,377
12
67
119

ok, i anwer fully tomorrow – Sergey Shustikov Jul 15 '14 at 06:12

Android: retrieve information from an internet site

8 Answers8