0

I'm using rtweet's function get_timeline to download tweets. However, some of the users I'm interested in have way more than the 3200 tweets you are allowed to download (some have around 47'000). There is the "retryonratelimit" argument, if you are downloading tweets based on words or hashtags, therefore I'm wondering whether there is a similar way to get more than 3200 tweets from one user?

bgf
  • 11
  • 3
  • 1
    you can write a simple for loop and the `max_id` argument to go further and further back in time. If you have a specific question, usually you would provide your data and code you tried so far. See [here](https://stackoverflow.com/a/5963610/5028841). I think that is why you were voted down. – JBGruber Apr 28 '19 at 23:05
  • Thank for your answer @JBGruber! Normally thats what you do: zanetti <- get_timeline("@zac1967", n=3200) But how can I include this function in a for loop? – bgf Apr 28 '19 at 23:40

1 Answers1

0

The documentation - see ?get_timeline - includes a link to the Twitter developer documentation for GET statuses/user_timeline . The R function is just a wrapper for this.

If you then follow the link to Working with timelines, you'll find an explanation of the max_id parameter.

The basic approach then is:

  1. get the first 3200 tweets
  2. get the earliest status ID using something like min(as.numeric(zanetti$status_id))
  3. run get_timeline again setting max_id = ID where ID is the ID from step 2

Note: I just tried this using my own timeline and only 40 tweets were returned by step 3. So you may also have to wait an appropriate amount of time to avoid rate limits. And be aware that Twitter basically does all it can to prevent you from requesting large amounts of data via the API - at the end of the day, what you want may not be possible.

neilfws
  • 32,751
  • 5
  • 50
  • 63
  • Thank you. I found this solution as well and tried it already. Unfortunately, it does not work - even after waiting considerably more than 15min it returned not a single tweet. Any chance you know of a different package I could use or does this restrictions apply to all of them? – bgf Apr 29 '19 at 00:15
  • The restrictiions are imposed by the Twitter API. – neilfws Apr 29 '19 at 00:25
  • 1
    I think you are right. Pretty sure it used to work before but it seems they have changed it. https://github.com/mkearney/rtweet/issues/278 – JBGruber Apr 29 '19 at 11:29
  • The timelines APIs have always hard a hard maximum limit of 3200 Tweets. If you wanted to get more Tweets by an individual user, you could use the premium search APIs, but I do not know whether rtweet supports those yet. – Andy Piper Apr 29 '19 at 11:52