Using twitter4j to search through more than 100 queries

Question

I am trying to create a program that searches a query from twitter. The problem I am having is that the API returns only a 100 result queries and when I try to retrieve more it keeps giving me the same results again.

        User user = twitter.showUser("johnny");
        Query query = new Query("football");
        query.setCount(100);
        query.lang("en");


        int i=0;

    try {     

                QueryResult result = twitter.search(query);
                for(int z = 0;z<2;z++){
                for( Status status : result.getTweets()){

                    System.out.println("@" + status.getUser().getScreenName() + ":" + status.getText());



                        i++;

                }

    }

The program will print me 200 results relating to the query "football", but instead of giving me 200 different results it prints a 100 results twice. My end results should be that I can print as many different results as the rate limit allows. I have seen programs that return more than 100 responses for a specific user, but I haven't seen something that can return more than a 100 responses for a unique query like "football".

@timrau the answer for that post was quite confusing when trying to relate to my code and wanted to know if there was another way to address the issue relating to my code. — shah, Apr 04 '16 at 16:29

FeanDoe · Accepted Answer · 2016-04-05T17:36:51.373

To get more than 100 results on a search Query you need to call to the next iteration of the Query.

    Query query = new Query("football");
    QueryResult result;
    int Count=0;
    do {
        result = twitter.search(query);
        List<Status> tweets = result.getTweets();
        for (Status tweet : tweets) {
          System.out.println("@" + tweet.getUser().getScreenName() + ":" + tweet.getText());
            Count++;
        }
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
    while ((query = result.nextQuery()) != null);
    System.out.println(Count);
    System.exit(0);

I just tested it and got 275 tweets, keep in mind this from the documentation:

The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.

And:

Before getting involved, it’s important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.

is it normal that the results change? Sometimes i get 108 tweets and other times I get 12. — shah, Apr 05 '16 at 16:56
Yep, its normal: "Before getting involved, it’s important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead." — FeanDoe, Apr 05 '16 at 17:11
I am trying to run the program multiple times in one call, but it keeps printing the same results. In essence I want to be able to keep making calls to the API till I hit the ratelimit. However, right now it will hit the ratelimit but keep printing the same set of results over and over again. Is there away around this? — shah, Apr 07 '16 at 13:12

Using twitter4j to search through more than 100 queries

1 Answers1