Parse dynamically loading (by scroll) page using JSOUP

Question

I am trying to count number of apps for a specific string. Like Flash Light, and here is the link that i am using to load page in jsoup,

Jsoup.connect("https://play.google.com/store/search?q=Flash+Light&c=apps&gl=us&hl=en")

Problem is that it only return 20 apps but there are more than 100 apps results when i open it in browser and scroll down. When i monitored closely i found out that for first time PalyStore shows 20 results rest of the results are fetched on scrolling.

Can anyone please tell me how to handle that?

Also i just want to count number of results if there is any other way that would be great too.

Possible duplicate of [Getting Jsoup to support dynamically generated html by JavaScript](http://stackoverflow.com/questions/12625043/getting-jsoup-to-support-dynamically-generated-html-by-javascript) — Frederic Klein, Oct 11 '16 at 14:00

score 1 · Answer 1 · answered Oct 11 '16 at 13:23

1

Jsoup cannot process dynamically loaded content. You need a different set of tools, like htmlunit.

answered Oct 11 '16 at 13:23

Andrei Volgin

40,755
6
49
58

htmlunit is too slow, and time performance is the primary criteria. – Zia ur Rehman Oct 11 '16 at 15:48
@ZiaurRehman it is still the best option for Java only solutions. Otherwise try phantomjs. (similar problem for google image search solved with phantomjs: http://stackoverflow.com/a/39054195/1661938) – Frederic Klein Oct 12 '16 at 07:54
@ZiaurRehman: Your comment does not change my answer - you cannot do it with Jsoup. – Andrei Volgin Oct 12 '16 at 15:58

Parse dynamically loading (by scroll) page using JSOUP

1 Answers1