I have an academic research API for Twitter and have been using the Twarc Python library to scrape tweets.
For actual tweet scraping it works really well. However, when scraping the followers of accounts it seems incredibly slow.
My understanding is the rate limit for queries is 15 queries every 15 minutes with 1,000 followers pulled per query. That should lead to a maximum of around 60,000 followers pulled in an hour. However, the actual speeds seem much lower.
For instance, scraping followers for an account with just under 15,000 followers took 5 hours yesterday (instead of the best scenario 15 minutes).
The beneath is the code I have been using for this.
Is there anything with my code that may be causing the slow speeds? Is there a better (faster) way to pull the followers of accounts using Python and the Twitter API?
Thanks to anyone able to help.
from twarc.client2 import Twarc2
from twarc.expansions import ensure_flattened
twarc = Twarc2(#VARIOUS API LOGIN INFO HERE#)
search = twarc.followers("TwitterAccountToScrape",max_results=50,user_fields=['id'])
followers = []
for page in search:
for follower in ensure_flattened(page):
followers.append(follower)
