save the session using JSOUP to parse html Android

Question

I need to parse and store data from this 2 tables in this link, after using "Inspect element" the network show me the request of tables :

• Request 1 to get the top table : request1
• Request 2 to get the bottom table : request2

What I did:

1. I send a get request to a link
2. I store cookie and user agent
3. I send another request to request1 with cookie and userAgent
4. I send another request to request2 with cookie and userAgent

it's not working, even If I try to open those 2 request in my navigator, it's not working, so How I can get data from those tables, and store data in java list ? this what I did :

Button Rechercher = (Button) findViewById(R.id.button);

        Rechercher.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                if (android.os.Build.VERSION.SDK_INT > 9) {
                    StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
                    StrictMode.setThreadPolicy(policy);
                    wv2.getSettings().setLoadsImagesAutomatically(true);
                    wv2.getSettings().setJavaScriptEnabled(true);
                    wv2.setScrollBarStyle(View.SCROLLBARS_INSIDE_OVERLAY);
                    wv2.loadUrl(requestCTM);
                    TestHtmlParse parse = new TestHtmlParse();
                    parse.toJava(requestCTM, wv2.getSettings().getUserAgentString());

                }
            }
        });

Classe Implementation

  public class TestHtmlParse {
    private Connection.Response response;

    public ArrayList<String> toJava(String url, String useragent) {
    ArrayList<String> downServers = new ArrayList<>();
        Date date = new Date();
        String URL1 = "http://billetterie.ctm.ma/site/select_voyages_aller?datev=03-06-2016&agen_dep=001&agen_dest=002&nbp=1&dater=03-06-2016&aller=AS&tpay=MA&q=1&_search=false&nd="+date.getTime()+"&rows=30&page=1&sidx=&sord=";

        Document doc = null;
        try {
            if (android.os.Build.VERSION.SDK_INT > 9) {
                StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
                StrictMode.setThreadPolicy(policy);
//first request
                response = Jsoup.connect(url).userAgent(useragent).timeout(10000).method(Connection.Method.GET).execute();
                Map<String, String> cookies = response.cookies();
// second request with cookie
                doc = Jsoup.connect(URL1).userAgent(useragent).timeout(10000).cookies(cookies).get();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

Answer 1

It won't work. The table you are looking for are loaded asynchronously via AJAX. Jsoup does not support AJAX, it only gets for you the plain html that was not manipulated by JavaScript.

Your best bet would be to use headless browsers like HtmlUnit or Selenium.