Getting html of a page in android

Question

Possible Duplicate:
Is it possible to get the HTML code from WebView

I'm trying to make an app which needs to get the html code from a website, and go through the code to look for images. Where I'm at now gives me a html code back in the emulator, but it is a different code I get when I open the source code for the website on my computer.

public String getInternetData(String adresse) throws Exception{
    BufferedReader in = null;
    String data = null;
    try{
        HttpClient client = new DefaultHttpClient();
        URI website = new URI(adresse);
        HttpGet request = new HttpGet();
        request.setURI(website);
        HttpResponse response = client.execute(request);
        in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
        StringBuffer sb = new StringBuffer("");
        String l = "";
        String nl = System.getProperty("line.separator");
        while ((l = in.readLine()) !=null){
            sb.append(l + nl);
        }
        in.close();
        data = sb.toString();
        return data;
    }finally{
        if (in != null){
            try{
                in.close();
                return data;
            }catch (Exception e){
                e.printStackTrace();
            }
        }
    }
}

The code I get in the emulator ends with something about enabling of JavaScript and Cookies being required. If this is my problem, how do I go about solving it?

Any help would be much appreciated!

Answer 1

1. Sometimes problem is in UserAgent(a server can send simplified/mobile version of a page watching your useragent string). Try to use the same useragent in yourcode as your browser
2. Try HtmlUnit "headless browser" framework to execute javascript.

Answer 2

see Aymon Fournier's Answer on this SO thread,

private String getDownloadButtonOnly(String url){
    HttpGet pageGet = new HttpGet(url);

    ResponseHandler<String> handler = new ResponseHandler<String>() {
        public String handleResponse(HttpResponse response) throws ClientProtocolException, IOException {
            HttpEntity entity = response.getEntity();
            String html; 

            if (entity != null) {
                html = EntityUtils.toString(entity);
                return html;
            } else {
                return null;
            }
        }
    };

    pageHTML = null;
    try {
        while (pageHTML==null){
            pageHTML = client.execute(pageGet, handler);
        }
    } catch (ClientProtocolException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

        Pattern pattern = Pattern.compile("<h2>Direct Down.+?</h2>(</div>)*(.+?)<.+?>", Pattern.DOTALL);
        Matcher matcher = pattern.matcher(pageHTML);
        String displayHTML = null;
        while(matcher.find()){
            displayHTML = matcher.group();
        }

    return displayHTML;
}

    @Override
    public void customizeWebView(final ServiceCommunicableActivity activity, final WebView webview, final SearchResult mRom) {
        mRom.setFileSize(getFileSize(mRom.getURLSuffix()));
        webview.getSettings().setJavaScriptEnabled(true);
        WebViewClient anchorWebViewClient = new WebViewClient()
        {

            @Override
            public void onPageStarted(WebView view, String url, Bitmap favicon) {
                super.onPageStarted(view, url, favicon);
                String downloadButtonHTML = getDownloadButtonOnly(url);
                if(downloadButtonHTML!=null && !url.equals(lastLoadedURL)){
                    lastLoadedURL = url;
                    webview.loadDataWithBaseURL(url, downloadButtonHTML, null, "utf-8", url);
                }
            }