4

I am trying to get the follower list of a given Twitter account. To do so, I manage to connect to my account using Selenium and PhantomJS (v2.1.1). Then, since Twitter is using infinite scrolling, I would like to implement the following algorithm:

1/ Count the number of html elements with class attribute equals ProfileCard-screennameLink 

2/ Repeat:
    -Scroll down
    -Count the number of html elements with class attribute equals ProfileCard-screennameLink
Until the number of elements does not change

My problem is that the scrolling function does not work. Indeed, when I am running 

    elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
    print("c", elemsCount)

    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(10)
    elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
    print("c (new)", elemsCount)

There is no change in the number of retrieved HTML elements (neither in the height of the page).. It looks like the scrollTo function has no impact. Do you have any explanation or workaround to solve my problem please?

ou_ryperd
  • 2,037
  • 2
  • 18
  • 23
Yoann Pitarch
  • 139
  • 1
  • 13
  • Why aren't you using any of the numerous twitter modules for Python, such as [Tweepy](http://www.tweepy.org/)? – asongtoruin May 15 '17 at 16:02
  • One part of my project is using Tweepy (to get the current trends) but the point is that due to the API rate limit and since I would like to collect the followers of quite numerous accounts, using Tweepy is not relevant to me in this scenario. – Yoann Pitarch May 15 '17 at 16:05
  • If twitter will rate limit you via their API, you can bet they won't want you harvesting that much data from their website either. – asongtoruin May 15 '17 at 16:43
  • Thanks for your comments but that's not the point. I do succeed when using Safari as the web driver for Selenium. My problem is when using PhantomJS instead of Safari (and I do need to use PhantomJS since the server I plan running my code on has not installed server X). – Yoann Pitarch May 15 '17 at 17:02
  • I am facing the same issue, did you find a solution @YoannPitarch ? – A. Attia Jul 10 '17 at 00:39
  • @A.Attia Unfortunately no... Sorry – Yoann Pitarch Jul 11 '17 at 05:39

0 Answers0