How can I fetch only a part of a webpage from the net rather than downloading the whole page

Question

At the moment I am using a third party library called Android Query. I use it to download the webpage, then I parse the html for the bits that I want. These bits are mostly in one small part of the page, so the rest is discarded.

each page is about 100kb and I am fetching 200-300 pages which tastes a while especially on a slow connection.

Is there any method or library to allow m e to fetch a certain div?

The pages i am fetching are from google play market.

example code i am using

String url = "https://play.google.com/store/apps/details?id=com.touchtype.swiftkey";

aq.ajax(url, String.class, new AjaxCallback<String>() {

        @Override
        public void callback(String url, String html, AjaxStatus status) {
                parseHtml(html);
        }

});

edit: if is it not possible, is there a light weight version of Google play pages that I can access and download?

Look at http://stackoverflow.com/q/7556286/1626878 – Sahil Mahajan Mj Nov 16 '12 at 08:22 — Sahil Mahajan Mj, Nov 16 '12 at 08:22

score 0 · Answer 1 · edited May 23 '17 at 12:12

Looking at the other answer here: Is there a way to download partial part of a webpage, rather than the whole HTML body, programmatically?

It looks like there are ways to do it, but you'd need to know the bytes range. This would be very error prone as the content could easily change over time if google changes different parts of the page.

You could setup your own web server to sit in between your app and google play and return the data you query.

There is a Hacker News thread about this: https://news.ycombinator.com/item?id=4634259

Apparently Apple has a JSON API, but Google does not.

Here are two server side libraries you could use to make the task easier:

https://github.com/chadrem/market_bot (Ruby)
https://github.com/pastfuture/MarketBot (PHP)

You could return only the data you need from your web app, making it as slim as possible. You could also cache it on your server so that you don't have to hit google for every request. Send several app ids in one request to minimize round trips.

How can I fetch only a part of a webpage from the net rather than downloading the whole page

1 Answers1