Is there a way to access html data of a website using webview in android?

Question

Setup: An android app that is supposed to show you your shopping behavior and sum up total online expenses I would like to analyse data that is displayed on a website, which I DON'T own using webview in my android app.

Problem: So i.e. the app should be capable of detecting what a user has in his "shopping cart", and when he presses the button to "buy" the products.

I read about JavaScriptInterfaces, however I really don't know anything about javascript and how the whole process of analysing should be constructed. If you could send me a few examples, or explain the process I would be very happy!

Answer 1

1) Implementation of a click detection on a button do the following:

Let me provide you a full example. For your the website http://store.nike.com/ch/de_de/pd/mercurial-superfly-v-tech-craft-2-herren-fussballschuh-fur-normalen-rasen/pid-11229711/pgid-11626158

  {
        ...

    webview.getSettings().setJavaScriptEnabled(true);
    webview.getSettings().setDomStorageEnabled(true);
    webview.addJavascriptInterface(new MyJavaScriptInterface(this), "ButtonRecognizer");

    webview.setWebViewClient(new WebViewClient() {
        @Override
        public void onPageFinished(WebView view, String url) {
            loadEvent(clickListener());
        }

        private void loadEvent(String javascript){
            webview.loadUrl("javascript:"+javascript);
        }

        private String clickListener(){
            return getButtons()+ "for(var i = 0; i < buttons.length; i++){\n" +
                    "\tbuttons[i].onclick = function(){ console.log('click worked.'); ButtonRecognizer.boundMethod('button clicked'); };\n" +
                    "}";
        }

        private String getButtons(){
            return "var buttons = document.getElementsByClassName('add-to-cart'); console.log(buttons.length + ' buttons');\n";
        }
    });

    webview.loadUrl("http://store.nike.com/ch/de_de/pd/mercurial-superfly-v-tech-cra\u200C\u200Bft-2-herren-fussballschuh-fur-normalen-rasen/pid-11229711/pgid-11626158");

    ...
}


class MyJavaScriptInterface {

    private Context ctx;

    MyJavaScriptInterface(Context ctx) {
        this.ctx = ctx;
    }

    @JavascriptInterface
    public void boundMethod(String html) {
        new AlertDialog.Builder(ctx).setTitle("HTML").setMessage("It worked")
                .setPositiveButton(android.R.string.ok, null).setCancelable(false).create().show();
        }
    }

That'll change the onClick for the button to what you need.

Getting the elements by class (getElementsByClassName()) worked as expected, however not: getElementById(). Furthermore, it might be necessary to replace getElementsByClassName() by getElementsByName() such as for example on this website: https://www.digitec.ch/de/s1/product/lexon-flip-wecker-3522142. Where you would put getElementsByName('AddProductToCart')

Info: the \n and the weird combination of strings inside the clickListener method are set like this, because of the IDE. ( ";" are causing problems inside a string).

---

2) Implementation of a website data analysis such as detecting what a customer puts inside his shopping cart:

(The code takes website specific parts which are defined by tags/names etc and outputs them inside the Android Monitor). You have to define the tags and names for every website. To get them, simply have a look at the html code of the website.

import android.content.Context;
import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.webkit.JavascriptInterface;
import android.webkit.WebView;
import android.webkit.WebViewClient;

public class ShoppingCartAnalyzerActivity extends AppCompatActivity {


    String adidas_tag = "product";
    String nike_tag = "ch4_cartItem";
    String zalando_tag = "z-coast-fjord_article";
    String digitec_tag = "item-description";

    String adidas_shoppingcart_url = "https://www.adidas.ch/on/demandware.store/Sites-adidas-CH-Site/de_CH/Cart-Show";
    String nike_shoppingcart_url = "https://secure-store.nike.com/ch/checkout/html/cart.jsp?";
    String zalando_shoppingcart_url = "https://www.zalando_tag.ch/cart/";
    String digitec_shoppingcart_url = "https://www.digitec.ch/";

    WebView webview;
    private String[] shoppingCartItems;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        webview = new WebView(this);
        setContentView(webview);


        webview.getSettings().setJavaScriptEnabled(true);
        webview.getSettings().setDomStorageEnabled(true);
        webview.addJavascriptInterface(new MyJavaScriptInterface(this), "ShoppingCartAnalyser");

        webview.setWebViewClient(new WebViewClient() {
            @Override
            public void onPageFinished(WebView view, String url) {
                loadEvent(clickListener());
            }

            private void loadEvent(String javascript) {
                webview.loadUrl("javascript:" + javascript);
            }

            private String clickListener() {
                return getProducts() + "for(var i = 0; i < products.length; i++){\n" +
                        "\tconsole.log(products[i].innerText); ShoppingCartAnalyser.boundMethod(products[i].innerText,i,products.length); \n" +
                        "}";
            }

            private String getProducts() {
                return "var products = document.getElementsByClassName('" + zalando_tag + "'); console.log(products.length + ' products');\n";
            }
        });

        webview.loadUrl(zalando_shoppingcart_url);
    }

    private void displayItems() {
        for (String shoppingCartItem : shoppingCartItems) {
            Log.d("ShoppingCart", cleanString(shoppingCartItem));
        }
    }

    private String cleanString(String shoppingCartItem) {
        shoppingCartItem = shoppingCartItem.replace("Ändern", "");
        shoppingCartItem = shoppingCartItem.replace("Entfernen", "");
        shoppingCartItem = shoppingCartItem.replace("Bearbeiten", "");
        shoppingCartItem = shoppingCartItem.replace("Löschen", "");
        shoppingCartItem = shoppingCartItem.replace("icon-cart-minus", "");
        shoppingCartItem = shoppingCartItem.replace("icon-cart-plus", "");
        shoppingCartItem = shoppingCartItem.replace("Service + Zubehör", "");
        shoppingCartItem = shoppingCartItem.replaceAll("(?m)^[ \t]*\r?\n", "");
        return shoppingCartItem;
    }

    class MyJavaScriptInterface {

        private Context ctx;

        MyJavaScriptInterface(Context ctx) {
            this.ctx = ctx;
        }

        @JavascriptInterface
        public void boundMethod(String decodedShoppingCart, int i, int size) {
            if (i == 0) {
                shoppingCartItems = new String[size];
            }
            shoppingCartItems[i] = decodedShoppingCart;
            if (i == size - 1) {
                displayItems();
            }
        }
    }
}

Answer 2

You can refer to this to learn how to get HTML content of webview. As for parsing the HTML content, XmlPullParser to do it.