18

I understand the Twitter REST API has strict request limits (few hundred times per 15 minutes), and that the streaming API is sometimes better for retrieving live data.

My question is, what exactly are the streaming API limits? Twitter references a percentage on their docs, but not a specific amount. Any insight is greatly appreciated.

What I'm trying to do:

  • Simple page for me to view the latest tweet (& date / time it was posted) from ~1000 twitter users. It seems I would rapidly hit the limit using the REST API, so would the streaming API be required for this application?
redbird_is
  • 181
  • 1
  • 1
  • 4

2 Answers2

33

You should be fine using the Streaming API, unless those ~1000 users combined are tweeting more than (very) roughly 60 tweets per second at any moment.

Using the Streaming API endpoint statuses/filter with the follow parameter, you are allowed up to 5000 users. There is no rate limit except when the stream returns more than about 1% of the all tweets being tweeted at that moment. (60 tweets per second is 1% of the average rate of tweets, which is always fluctuating, so don't rely on that number.)

If your stream does go above the 1% threshold, you can detect this. (See the LIMIT notice.) Then you would use the REST API to find missed tweets.

Jonas
  • 3,969
  • 2
  • 25
  • 30
  • 1
    Upvote for the hint on limit to find out about amount of missed tweets due to the threshold. – rdoubleui Jan 23 '16 at 15:49
  • Ok cool this makes sense, thank you. So if I needed to monitor 5001 twitter users, I would hit a limit there as well? – redbird_is Jan 23 '16 at 19:46
  • hey @Jonas, do you have any source/documentation you can link for this? – Pega88 May 19 '17 at 19:35
  • Hi @Pega88 Twitter does not publish this. It's easy to find statistics by googling, nor is it difficult to measure this yourself. – Jonas May 20 '17 at 01:59
  • @Jonas, I would like to know that if I am using Twitter Streaming API to know about tweets related to a keyword, then for how long it can last – shekhar Jul 02 '19 at 12:30
  • @shekhar Here is Twitter's documentation for streaming https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html. Use the `track` parameter. I don't understand what you are referring to by "how long it can last." – Jonas Jul 02 '19 at 15:28
  • @Jonas sorry for not being clear, I mean that if I put my script on a cloud server like digital ocean, then without crashing for how many days it can run and store those keyword related tweets. – shekhar Jul 02 '19 at 15:47
  • 2
    @shekhar That depends on how you write your code. There are a variety of error conditions that you must code for. Read this to get an idea http://geduldig.github.io/TwitterAPI/faulttolerance.html – Jonas Jul 02 '19 at 20:09
  • @Jonas Thank you very much for your answer! But I am wondering if the 1% rule still holds for today. And if I use the streaming API to get the tweets posted in a predefined bounding box, could I get all the tweets posted in that bounding box if the 1% limit is not reached? – Bright Chang Sep 06 '20 at 03:57
  • @BrightChang The exact percentage that represents the "small random sample" is not published, but the percentage probably has not changed. Yes, you can use the bounding box to get all tweets provided the stream does not reach the limit. – Jonas Sep 06 '20 at 19:04
  • Hey @Jonas, sorry for digging on an old thread. I understand that the streaming API would only hit the limit when returning about 60 tweets a sec - which also is a rough number. Can I assume that I'd be fine if I stream tweets from like 2-3 profiles, publishing frequency is way less than 60/sec and that too with filtering retweets? I'm pretty sure I'd be but just wanted to confirm :) appreciate any suggestion... – P S Solanki May 29 '21 at 15:21
  • Sounds okay to me. If I understand you... you will have 3 streams, each authenticated with different credentials. – Jonas May 29 '21 at 15:28
  • appreciate the response. Is that possible to have one stream (filter) and get it to pull tweets from like 2 or 3 users? @Jonas – P S Solanki May 29 '21 at 15:30
  • Use the `follow` parameter described here https://developer.twitter.com/en/docs/twitter-api/v1/tweets/filter-realtime/api-reference/post-statuses-filter – Jonas May 29 '21 at 15:37
  • Here is an interesting observation on dev portal, [THIS](https://pasteboard.co/K49A7kv.png) is a snapshot of filter stream endpoint docs. It says to right, rate limit = `50 per 15 min`. What does that mean? (btw I think i created a project which is on v2 endpoints). not sure if that makes a difference. @Jonas – P S Solanki May 29 '21 at 15:41
  • 1
    Each stream connection is one request. You can find similar docs for v2 endpoints on developer.twitter.com. Start there. – Jonas May 29 '21 at 15:49
  • ahhh so it means upto 50 streams in a 15 min window - that's pretty good. Also the link to `start here` didn't render but no worries. I appreciate the help :) – P S Solanki May 29 '21 at 15:52
  • That was not a link. A link is https://developer.twitter.com/ – Jonas May 29 '21 at 19:42
7

Twitter simply will not allow multiple streams from one registered app/account. Doing so will result in the older one being closed.

Also too many connection tries are not allowed as well and will result in a user being blocked.

Reference docs: Public Streaming API (outdated)

rdoubleui
  • 3,554
  • 4
  • 30
  • 51
  • I think a single stream would work fine in this situation. – Jonas Jan 23 '16 at 15:03
  • 1
    That's right, a single public stream is the way to go in this case. I was referring to the op asking for the restrictions of the streaming api. – rdoubleui Jan 23 '16 at 15:12
  • 3
    Thanks - good to know about the multiple connection tries. This means that I shouldn't connect to the streaming API feed every time I refresh / open my page that displays the tweets, correct? I should have a standalone file continuously running that is maintaining a connection to the streaming API? – redbird_is Jan 23 '16 at 19:48
  • Exactly, you need to to prevent being blocked. – rdoubleui Jan 23 '16 at 20:54
  • The Public Streaming API link redirects to /docs now. The last version that had useful content dates from [Sep 2017](https://web.archive.org/web/20170920135622/https://dev.twitter.com/streaming/public). – Dan Dascalescu Mar 21 '19 at 07:58
  • Thanks for the hint, my answer is from Jan 2016, and tbh I haven't worked on projects like that since then, so, sorry when this is outdated, but it's hard to maintain your answers over several years. In general I think possibilities regarding usage of these APIs have shifted. – rdoubleui Mar 22 '19 at 09:13