Jsoup parse dynamically loading webpage in Java

Question

import java.io.IOException;
import java.util.ArrayList;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Node;
import org.jsoup.select.Elements;


public class listGrabber {
    public static void main(String[]args) {
        try {
            Document doc = Jsoup.connect("https://play.google.com/store/apps/category/GAME_ACTION/collection/topselling_free").get();
            int count = 0;
            Elements elements;
            String url;
            ArrayList<String> list = new ArrayList<>();
            do{
                elements = doc.select("a[class^=title]").get(count).select("a[class^=title]");

                url = "";
                url = elements.attr("abs:title").replaceAll("https://play.google.com/store/apps/category/GAME_ACTION/collection/","");
                url = url.replaceAll("®|™","");
                url = url.replaceAll("[(](.*)[)]","");
                list.add(url);
                System.out.println(url);
                count++;
            }while (url!="" &&url!=null);
            // String divContents =
            // doc.select(".id-app-orig-desc").first().text();
            // elements.remove("div");
        } catch (IOException e) {

        }
    }
}

As you can see above, I'm trying to grab a list of words from https://play.google.com/store/apps/category/GAME_ACTION/collection/topselling_free

The google play store page loads more elements every time you scroll to the bottom of the page.

My program will grab the first 40ish elements that show up but since jsoup doesn't load the rest of the webpage that loads dynamically, I can't grab any of the elements beyond the first 40.

Furthermore, if you scroll on the page to game #300, a Show More button appears, I'd also like to parse the elements beyond the show more button.

Is there any way for Jsoup to parse all the elements that would dynamically load on the page?

Answer 1

EDIT - After few comments from the OP, I understood exectly what he wants to acheive. I've changed a bit my original solution and tested it.

You can do it with JSOUP. After the first page, getting the next one requiers you to sen a post request with some headers. The headers contains (among other) the start number and how many records to get. If you send an illegel number (i.e. you ask the page that contains game number 700 but the results contain only 600 games), you get the first page again. You can loop thru the pages, until you get a result that you already have.
Sometimes the server returns 600 results and sometimes only 540, I could not figure why.
The code for that is -

import java.util.regex.Pattern;
import org.jsoup.Connection;
import org.jsoup.Connection.Method;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;

public class HelloWorld {

public static void main(String[] args) {

    Connection.Response res = null;
    Document doc = null;
    Boolean OK = true;
    int start = 0;
    String query;
    ArrayList<String> tempList = new ArrayList<>();
    ArrayList<String> games = new ArrayList<>();
    Pattern r = Pattern.compile("title=\"(.*)\" a");

    try {   //first connection with GET request
        res = Jsoup.connect("https://play.google.com/store/apps/category/GAME_ACTION/collection/topselling_free")
                .method(Method.GET)
                .execute(); 
        doc = res.parse();
    } catch (Exception ex) {
        //Do some exception handling here
    }
    for (int i=1; i <= 60; i++) {    //parse the result and add it to the list
        query = "div.card:nth-child(" + i + ") > div:nth-child(1) > div:nth-child(3) > h2:nth-child(2) > a:nth-child(1)";
        tempList.add(doc.select(query).toString());
    }

    while (OK) {    //loop until you get the same results again
        start += 60;    
        System.out.println("now at number " + start);
        try {      //send post request for each new page
            doc = Jsoup.connect("https://play.google.com/store/apps/category/GAME_ACTION/collection/topselling_free?authuser=0")
                    .cookies(res.cookies())
                    .data("start", String.valueOf(start))
                    .data("num", "60")
                    .data("numChildren", "0") 
                    .data("ipf", "1")
                    .data("xhr", "1")
                    .post();
        } catch (Exception ex) {
            //Do some exception handling here
        }
        for (int i=1; i <= 60; i++) {    //parse the result and add it to the list
            query = "div.card:nth-child(" + i + ") > div:nth-child(1) > div:nth-child(3) > h2:nth-child(2) > a:nth-child(1)";
            if (!tempList.contains(doc.select(query).toString())) {
                tempList.add(doc.select(query).toString());
            } else {    //we've seen these games before, time to quit
                OK = false;
                break;
            }               
        }   
    }
    for (int i = 0; i < tempList.size(); i++) {    //remove all redundent info.
        Matcher m = r.matcher(tempList.get(i));
        if (m.find()) {
            games.add(m.group(1));
            System.out.println((i + 1) + " " + games.get(i));
        }           
    }
}
}

The code can be further improved (like handling all the lists at a seperate method), so it's up to you.
I hope this does the work for you.