How to get tweets older than few months?

Question

I am retrieving tweets using LINQ to Twitter 4.1.0 with particular hashtag. But the problem that I am facing is that i am getting only 600 tweets. I want to get tweets that are older than few months. Can anyone suggest me, how to do this?

 [List<Search> searchResultsList = new List<Search>();
        ulong lastId = 0;
        var context = new TwitterContext(auth);
        int count = 0;
        int maxNumberToFind = 1000;
        var dateFrom = DateTime.Now.AddDays(-20);

        var searchResults =
                         (from search in context.Search
                          where search.Type == SearchType.Search &&
                                search.Query == Query &&
                                search.SinceID == 2016-12-11 &&                                
                                search.Count == 150 &&
                                search.ResultType == ResultType.Recent &&
                                search.IncludeEntities == true
                          select search).ToList();][1]

I have no idea, but have you tried to change search.SinceID? — hellogoodnight, Jan 11 '17 at 14:17
Are you trying to specify a date range for which to get the tweets? — big_water, Jan 11 '17 at 14:17
@big_water: Actually I don't have any idea about daterange. But I was trying to do that with until. but it didn't worked for me. :( — Ranjana, Jan 11 '17 at 14:22
@hellogoodnight: dateFrom is extra in my code. And by changing search.SinceID also, it id not working. — Ranjana, Jan 11 '17 at 14:24
@Ranjana, SinceId takes in a "ulong" data type which should be an ID. 2016-12-11 is a date (actually not even that the way you have it typed). Does this code compile? — big_water, Jan 11 '17 at 14:27
@big_water: Yaa I got your point. Yaa my code compiled, but its returning incorrect value of SinceID. — Ranjana, Jan 11 '17 at 14:38

score 2 · Answer 1 · edited Jun 20 '20 at 09:12

I would suggest looking at the documentation It looks like you can specify an "Until" date, and a MaxId:

Until: Tweets up to this date, YYYY-MM-DD. (string)

MaxID: Return tweets prior to or equal to this ID. (ulong)

I would query for an "Until" date and get the last tweet using LINQ's "LastOrDefault()". Then do another query using the ID from that object as the MaxID of your next query. This should get you all tweets prior to a certain date as you asked for.

It's also important to note this parameter:

Count Number of tweets to retrieve for each page. Max is 100. (int)

This might have something to do with the limited number of tweets you are receiving.

EDIT: This post by Joe Mayo (the creator of LinqToTwitter) might also be helpful to you. Here is some code (disclaimer: I haven't tested this, but you get the idea):

List<Search> searchResultsList = new List<Search>();
    var context = new TwitterContext(auth);
    var maxCount = 100;
    var untilDate = new DateTime(2016,12,11);

    var lastTweetInRange =
                     (from search in context.Search
                      where search.Type == SearchType.Search &&
                            search.Query == Query &&                               
                            search.Count == maxCount &&
                            search.Until == untilDate &&
                            search.IncludeEntities == true
                      select search).LastOrDefault();

    var tweetsInRange = (from search in context.Search
                        where search.Type == SearchType.Search &&
                                search.Query == Query &&                               
                                search.Count == maxCount &&
                                search.MaxId == lastTweetInRange.id &&
                                search.IncludeEntities == true
                        select search).ToList();

@Ranjana, I've updated with some code. I haven't tested it but it should help visualize what I was describing :) — big_water, Jan 11 '17 at 15:00
Thanks for your response. I am getting error at Until. https://i.stack.imgur.com/NZ6mU.png — Ranjana, Jan 11 '17 at 15:11
@Ranjana, can you update your question with the message & stacktrace of the runtime error? — big_water, Jan 11 '17 at 15:19
@big_water: It worked for me.. But I am getting data of only 10 days back. — Ranjana, Jan 12 '17 at 15:22

Joe Mayo · Answer 2 · 2017-01-13T05:01:34.630

You might want to also consider a paged search. Here's a demo:

    static async Task DoPagedSearchAsync(TwitterContext twitterCtx)
    {
        const int MaxSearchEntriesToReturn = 100;

        string searchTerm = "twitter";

        // oldest id you already have for this search term
        ulong sinceID = 1;

        // used after the first query to track current session
        ulong maxID; 

        var combinedSearchResults = new List<Status>();

        List<Status> searchResponse =
            await
            (from search in twitterCtx.Search
             where search.Type == SearchType.Search &&
                   search.Query == searchTerm &&
                   search.Count == MaxSearchEntriesToReturn &&
                   search.SinceID == sinceID
             select search.Statuses)
            .SingleOrDefaultAsync();

        combinedSearchResults.AddRange(searchResponse);
        ulong previousMaxID = ulong.MaxValue;
        do
        {
            // one less than the newest id you've just queried
            maxID = searchResponse.Min(status => status.StatusID) - 1;

            Debug.Assert(maxID < previousMaxID);
            previousMaxID = maxID;

            searchResponse =
                await
                (from search in twitterCtx.Search
                 where search.Type == SearchType.Search &&
                       search.Query == searchTerm &&
                       search.Count == MaxSearchEntriesToReturn &&
                       search.MaxID == maxID &&
                       search.SinceID == sinceID
                 select search.Statuses)
                .SingleOrDefaultAsync();

            combinedSearchResults.AddRange(searchResponse);
        } while (searchResponse.Any());

        combinedSearchResults.ForEach(tweet =>
            Console.WriteLine(
                "\n  User: {0} ({1})\n  Tweet: {2}",
                tweet.User.ScreenNameResponse,
                tweet.User.UserIDResponse,
                tweet.Text));
    }

I wrote a blog post a while back to explain generally how this works:

Working with Timelines with LINQ to Twitter

It's a little old and doesn't include the async syntax, but does explain the SinceID, MaxID, and techniques. Twitter also has good documentation, explaining the how and why of their paging strategy:

Working with Timelines (Twitter)

With that said, the Twitter API does limit how far back you can go with searches. In The Search API, Best Practices section, they describe that they only go back 6 to 9 days.

Yup thnx a lot.. But I am getting tweets of 10 days back only.. Not more than that.. Coz this api does not gives flexibility to get historic data.. — Ranjana, Jan 12 '17 at 15:21
To be clear - the limitation is from the Twitter API and not this or any other library. — Joe Mayo, Jan 12 '17 at 19:57

How to get tweets older than few months?

2 Answers2