0

I want to parse a Twitter list (e.g. https://twitter.com/spdbt/lists/spd-bundestagsabgeordnete/members) using JSoup. My problem is, that the page is dynamic, i.e. that I only get the first 20 results from the page. Is there any way JSoup can fetch the whole page?

Currently, my codes looks as follows:

Document doc = Jsoup.connect(listAdress).get();
Elements usernames = doc.select(".username.js-action-profile-name");
Elements realNames = doc.select(".fullname.js-action-profile-name");
// iterate over usernames and realNames and do something

Thanks in advance!

pes04
  • 1
  • 3
  • I do not think this is possible : [More info](http://stackoverflow.com/questions/25749309/using-jsoup-to-parse-a-dynamic-page) – Yassin Hajaj Nov 23 '15 at 19:22

2 Answers2

0

Some work around to achieve this

  • Launch browser with above URL using Selenium
  • Load page fully
  • get the page source using Selenium method.
  • Pass this content to JSOUP
  • Parse it.

Logic

WebDriver driver = new FirefoxDriver();
driver.get("https://twitter.com/spdbt/lists/spd-bundestagsabgeordnete/members")
//some logic to scroll or you do it manually
String pageContent = driver.getPageSource();
Document doc = Jsoup.parse(pageContent);
//from here write your logic to get the required values
Santoshsarma
  • 5,627
  • 1
  • 24
  • 39
0

finally solved the problem by using a Twitter library, but thanks for your help.

pes04
  • 1
  • 3