5

I am trying to fetch tweets from Twitter timeline using the max_id field present in next_results as demonstrated here.(Twython search API with next_results)

For most of the queries I get the desired number(500+) of tweets but for some I get less than 200 tweets.

When looking into what is returned in 'next_results I found that after returning just 100 tweets, the response has next_results field missing. There cannot be just 200 tweets for the queried topic as I am searching for this topic when it is trending and on Twitter I can see more than 200 tweets.

Is anyone else also facing this issue. Is there any workaround ?

Below is the output where in the second response the 'next_results' field is missing.

TEST_PROGRAM >> Calling twitter to get tweets for  #TheWorstLieEver

{u'count': 100, u'completed_in': 0.066, u'max_id_str': u'432942833725497345', u'since_id_str': u'0', **u'next_results': u'?max_id=432540545630494719&q=%23TheWorstLieEver&count=100&include
_entities=1&result_type=mixed'**, u'refresh_url': u'?since_id=432942833725497345&q=%23TheWorstLieEver&result_type=mixed&include_entities=1', u'since_id': 0, u'query': u'%23TheWorstLieEver
', u'max_id': 432942833725497345L}

{u'count': 100, u'completed_in': 0.111, u'max_id_str': u'432540545630494719', u'since_id_str': u'0', u'refresh_url': u'?since_id=432540545630494719&q=%23TheWorstLieEver&result_type=mixe
d&include_entities=1', u'since_id': 0, u'query': u'%23TheWorstLieEver', u'max_id': 432540545630494719L}
Community
  • 1
  • 1
kundan
  • 1,278
  • 14
  • 27
  • Hey, did you find a solution? I've got the exact same problem – Colin Bernet Feb 13 '14 at 14:26
  • @ColinBernet It seems implementing own max_id navigation fetches more results. What it means is, when we get a response with no `next_results` we need to manually find the tweet with smallest tweet id and submit it as `max_id` in the next query. You can find its reference here:[link](https://github.com/ckoepp/TwitterSearch/issues/6).I tried this but it didnt fetch me any more tweets !! – kundan Feb 14 '14 at 05:45
  • @ColinBernet I got the answer. If you are looking for volume, you need to use Streaming API instead. Twitter search API focuses on relevance of query and hence might not return all tweets. [Twitter documentation] (https://dev.twitter.com/docs/using-search) says `Before getting involved, it's important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.` – kundan Feb 14 '14 at 17:49
  • @kundan: Since you found a solution, you should add it as an answer to your question and accept it. – Aaron Hill Mar 13 '14 at 18:50
  • @Aaron Thanks for pointing it. I will do it. – kundan Mar 16 '14 at 08:15

1 Answers1

2

I got the answer. If you are looking for volume, you need to use Streaming API instead. Twitter search API focuses on relevance of query and hence might not return all tweets. Twitter documentation says Before getting involved, it's important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.

kundan
  • 1,278
  • 14
  • 27