How can i retrive from website with high javascript in Android?

Question

http://www.biletix.com/search/TURKIYE/en#!subcat_interval:12/12/15TO19/12/15

I want to get data from this website. When i use jsoup, it cant execute because of javascript. Despite all my efforts, still couldnot manage.

enter image description here

As you can see, i only want to get name and url. Then i can go to that url and get begin-end time and location.

I dont want to use headless browsers. Do you know any alternatives?

Answer 1

Sometimes javascript and json based web pages are easier to scrape than plain html ones.

If you inspect carefully the network traffic (for example, with browser developer tools) you'll realize that page is making a GET request that returns a json string with all the data you need. You'll be able to parse that json with any json library.

URL is:

http://www.biletix.com/solr/en/select/?start=0&rows=100&fq=end%3A[2015-12-12T00%3A00%3A00Z%20TO%202015-12-19T00%3A00%3A00Z%2B1DAY]&sort=vote%20desc,start%20asc&&wt=json

You can generate this URL in a similar way you are generating the URL you put in your question.

A fragment of the json you'll get is:

....
 "id":"SZ683",
 "venuecount":"1",
 "category":"ART",
 "start":"2015-12-12T18:30:00Z",
 "subcategory":"tiyatro$ART",
 "name":"The Last Couple to Meet Online",
 "venuecode":"BT",
.....

There you can see the name and URL is easily generated using id field (SZ683), for example: http://www.biletix.com/etkinlik/SZ683/TURKIYE/en

------- EDIT -------

Get the json data is more difficult than I initially thought. Server requires a cookie in order to return correct data so we need:

• To do a first GET, fetch the cookie and do a second GET for obtain the json data. This is easy using Jsoup.
• Then we will parse the response using org.json.

This is a working example:

//Only as example please DON'T use in production code without error control and more robust parsing
//note the smaller change in server will break this code!!
public static void main(String[] args) throws IOException {
    //We do a initial GET to retrieve the cookie
    Document doc = Jsoup.connect("http://www.biletix.com/").get();
    Element body = doc.head();
    //needs error control 
    String script = body.select("script").get(0).html();

    //Not the more robust way of doing it ...
    Pattern p = Pattern.compile("document\\.cookie\\s*=\\s*'(\\w+)=(.*?);");
    Matcher m = p.matcher(script);
    m.find();
    String cookieName = m.group(1);
    String cookieValue = m.group(2);

    //I'm supposing url is already built
    //removing url last part (json.wrf=jsonp1450136314484) result will be parsed more easily 
    String url = "http://www.biletix.com/solr/tr/select/?start=0&rows=100&q=subcategory:tiyatro$ART&qt=standard&fq=region:%22ISTANBUL%22&fq=end%3A%5B2015-12-15T00%3A00%3A00Z%20TO%202017-12-15T00%3A00%3A00Z%2B1DAY%5D&sort=start%20asc&&wt=json";

    Document document = Jsoup.connect(url)
            .cookie(cookieName, cookieValue) //introducing the cookie we will get the corect results
            .get();
    String bodyText = document.body().text();

    //We parse the json and extract the data
    JSONObject jsonObject = new JSONObject(bodyText);
    JSONArray jsonArray = jsonObject.getJSONObject("response").getJSONArray("docs");
    for (Object object : jsonArray) {
        JSONObject item = (JSONObject) object;
        System.out.println("name = " + item.getString("name"));
        System.out.println("link = " + "http://www.biletix.com/etkinlik/" + item.getString("id") + "/TURKIYE/en");
        //similarly you can fetch more info ...
        System.out.println();
    }
}

I skipped the URL generation as I suppose you know how to generate it.

I hope all the explanation is clear, english isn't my first language so it is difficult for me to explain myself.