c# - Listing Friends -- Hitting auth failure mid-process

Question

So, I'm trying to the friends of all of someone's followers. Not recursively digging, just that single level.

I'm able to pull the followers just fine (and it looks like it can handle more than 75k followers, so I'm 90% sure I have the rate limits fine there).

But my code to pull the friends is throwing authentication errors after it returns 15 of the followers friends.

In other words, in this bit of code:

for (var i = 0; i < followerList.Count; i++)
{
  await followersFriends(followerList[i]);
}

I'm getting this error when i == 15:

LinqToTwitter.TwitterQueryException HResult=0x80131500 Message={"request":"/1.1/friends/ids.json","error":"Not authorized."} - Please visit the LINQ to Twitter FAQ (at the HelpLink) for help on resolving this error.

It does appear to completely get 15 lists, it's the 16th that throws it. I have the feeling that this is not a coincidence that that's the same as the rate limit, albeit an odd error if it is a rate limit.

I'll throw a check to see if we're on a multiple of 15 and do a 15 minute pause, but, on the chance that that doesn't work, does anyone have a theory as to what else could be going on here?

ETA: I did try redoing my authentication (Doing application based) on each run through followersFriends. Didn't help.

ETA2:

This is the new calling code:

for (var i = 0; i < followerList.Count; i++)
{
  if (i % 15 == 0 && i != 0) {
    Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - i = " + i + " and user = " + followerList[i]);
    PauseThatRefreshes("friends");
    RunMe();
  }

  await followersFriends(followerList[i]);
}

And here's the code being called:

private async Task followersFriends(ulong uid)
{
  var twitterCtx2 = new TwitterContext(m_foo);

  Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Start Friend Get for " + uid.ToString() + " ***");

  long cursor = -1;

  do
  {
    var friendList = await (from friend in twitterCtx2.Friendship where friend.Type == FriendshipType.FriendIDs && friend.UserID == uid.ToString() && friend.Cursor == cursor select friend).SingleOrDefaultAsync();

    if (twitterCtx2.RateLimitRemaining <= 2)
    {
      Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pausing A ***");
      Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause A: " +
        twitterCtx2.RateLimitRemaining.ToString());
      PauseThatRefreshes("friends");
    }

    if (friendList != null &&
      friendList.IDInfo != null &&
      friendList.IDInfo.IDs != null) {
      cursor = friendList.CursorMovement.Next;
      Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining: " + twitterCtx2.RateLimitRemaining.ToString());
      friendList.IDInfo.IDs.ForEach(id => Output(uid.ToString(), id.ToString()));
    }

    if (twitterCtx2.RateLimitRemaining <= 2)
    {
      Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pause B ***");
      Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause B: " + twitterCtx2.RateLimitRemaining.ToString());
      PauseThatRefreshes("friends");
    }
  } while (cursor != 0);

  Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Stop Friend Get for " + uid.ToString() + " ***");
  Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at stop: " + twitterCtx2.RateLimitRemaining.ToString());

  return;
}

ETA just for clarification, the code that is calling the followersFriends. Yes, I know, I didn't change my button name =D This is more about collecting the data and I prefer a form as opposed to a command line... I've no idea why. Anyways, neither here nor there:

        private async void button2_Click(object sender, EventArgs e)
    {

        var twitterCtx = new TwitterContext(m_foo);
        string[] header = { "Account,Follows" };
        File.WriteAllLines(@"C:\temp\followersFriends.txt", header);

        //List to store ids of followers of main account
        List<ulong> followerList = new List<ulong>();
        // get main accounts followers and put into an array
        long cursor = -1;
        Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Start Follower Get ***");
        do
        {

            var followers =
    await
    (from follower in twitterCtx.Friendship
     where follower.Type == FriendshipType.FollowerIDs &&
           follower.ScreenName == "[[redacted]]" && 
           follower.Cursor == cursor
     select follower)
    .SingleOrDefaultAsync();
            if (followers != null &&
                followers.IDInfo != null &&
                followers.IDInfo.IDs != null
                &&
                followers.CursorMovement != null)
            {
                cursor = followers.CursorMovement.Next;
                followers.IDInfo.IDs.ForEach(id =>
                  followerList.Add(id));
            }

            if (twitterCtx.RateLimitRemaining <= 2)
                PauseThatRefreshes("followers");
        } while (cursor != 0);
        Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Done Follower Get ***");

        for (var i = 0; i < followerList.Count; i++)
        {
            if (i % 15 == 0 && i != 0)
            {
                Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - i = " + i 
                    + " and user = " + followerList[i]);
                PauseThatRefreshes("friends");
                RunMe();
            }
            await followersFriends(followerList[i]);
        }
    }

Are you testing with the same `followerList` every time? Could it be that the problem is related to a specific call rather than the 16th request? — Camilo Terevinto, Jul 01 '18 at 00:39
Could you provide a bit more code? Which is the specific call you are using? — greyhame, Jul 01 '18 at 07:07
The "Not Authorized" does look strange and as you allude to, it seems like a rate limit situation. Can you update your question with the LINQ to Twitter query? - might help in reproducing the problem. — Joe Mayo, Jul 01 '18 at 21:16
Sure. Editing to add the code =) (Was hoping it was going to end up being a 'Oh, yeah, it happens because X') — John Eddy, Jul 02 '18 at 00:49

Joe Mayo · Accepted Answer · 2018-07-09T03:58:08.167

As indicated in your notes, a TwitterQueryException with a "Not Authorized" message throws when encountering the 1963joeym account. Inspecting this account, I can see that they've protected it. The concept of "protected" means that no one can see a person's account or related activity unless that person "authorizes" your account. Since they haven't authorized the account that is attempting to query their friends list, you receive the HTTP 401 Unauthorized, from Twitter. Thus, the reason for the "Not Authorized" message.

Although User entities have a bool Protected property, it isn't practical to check that in your case because you're only working with a list of IDs and don't have the full User instance available. In this situation, the only thing you can do is to handle the exception and recover from there. There are two places to handle this exception: (1) When querying the original friends list and (2) When querying the friend's friends list, shown below:

        var allFollowerIds = new List<ulong>();
        Friendship followers = null;
        long cursor = -1;
        do
        {
            try
            {
                followers =
                    await
                    (from follower in twitterCtx.Friendship
                     where follower.Type == FriendshipType.FollowerIDs &&
                           follower.UserID == "1039951639" &&// "15411837" &&
                           follower.Cursor == cursor &&
                           follower.Count == 500
                     select follower)
                    .SingleOrDefaultAsync();
            }
            catch (TwitterQueryException tqExUnauthorized) when (tqExUnauthorized.StatusCode == HttpStatusCode.Unauthorized)
            {
                Console.WriteLine("This user hasn't given your account permission to view their account.");
            }
            catch (TwitterQueryException tqe)
            {
                Console.WriteLine(tqe.ToString());
                break;
            }

            if (followers != null &&
                followers.IDInfo != null &&
                followers.IDInfo.IDs != null)
            {
                cursor = followers.CursorMovement.Next;

                allFollowerIds.AddRange(followers.IDInfo.IDs);
            }

        } while (cursor != 0);

        cursor = -1;
        foreach (var uid in allFollowerIds)
        {
            Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Start Friend Get for " + uid.ToString() + " ***");

            do
            {
                Friendship friendList = null;
                try
                {
                    friendList = await (from friend in twitterCtx.Friendship where friend.Type == FriendshipType.FriendIDs && friend.UserID == uid.ToString() && friend.Cursor == cursor select friend).SingleOrDefaultAsync();
                }
                catch (TwitterQueryException tqExUnauthorized) when (tqExUnauthorized.StatusCode == HttpStatusCode.Unauthorized)
                {
                    Console.WriteLine("This user hasn't given your account permission to view their account.");
                }

                if (twitterCtx.RateLimitRemaining <= 2)
                {
                    Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pausing A ***");
                    Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause A: " +
                      twitterCtx.RateLimitRemaining.ToString());
                    //PauseThatRefreshes("friends");
                }

                if (friendList != null &&
                  friendList.IDInfo != null &&
                  friendList.IDInfo.IDs != null)
                {
                    cursor = friendList.CursorMovement.Next;
                    Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining: " + twitterCtx.RateLimitRemaining.ToString());
                    //friendList.IDInfo.IDs.ForEach(id => Output(uid.ToString(), id.ToString()));
                }

                if (twitterCtx.RateLimitRemaining <= 2)
                {
                    Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pause B ***");
                    Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause B: " + twitterCtx.RateLimitRemaining.ToString());
                    //PauseThatRefreshes("friends");
                }
            } while (cursor != 0);

            Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Stop Friend Get for " + uid.ToString() + " ***");
            Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at stop: " + twitterCtx.RateLimitRemaining.ToString());
        }

Notice that I used a try/catch with a when clause, specifically for the 401 Unauthorized scenario, repeated below:

        catch (TwitterQueryException tqExUnauthorized) when (tqExUnauthorized.StatusCode == HttpStatusCode.Unauthorized)
        {
            Console.WriteLine("This user hasn't given your account permission to view their account.");
        }

For the friends list, this lets you handle just the Unauthorized portion and still protect for any other TwitterQueryException that doesn't fit that criteria. For the friend's friends list, you just need to handle only the Unauthorized exception and let the query continue to iterate on remaining friends.

Hmm. I'll try that in the morning, but I'm not sure I'm really nesting it (unless I'm misinterpreting...) Editing to add code here.... — John Eddy, Jul 02 '18 at 05:25
Added my code to the main post. Also: Thanks for the tips. Very useful. I should add: I've tried wayyyyy too many twitter libraries to count and this is the first one that I've had any success with so... good job =) — John Eddy, Jul 02 '18 at 05:32
I see that. I was only able to get "Not Authorized" by nesting the queries. This version results in a "Rate Limit" error, which it looks like you know how to handle. — Joe Mayo, Jul 02 '18 at 05:39
Odd. And nope, not authorized after 18 follower's friends pulls. It looks like it fully pulled user 18's list, except it did manage to pull more users than I see if I look at his account. I'm okay with some variance (I expected some due to hidden users who might match up to ones I'm allowed to see and the like, not to mention just normal data anomalies) so the number doesn't worry me too much. — John Eddy, Jul 03 '18 at 03:17
So, I ran a fiddler trace to see what happens. I kept thinking that this might be what the problem was, but, at the same time no, it couldn't be. The user it's stopping on is 1963joeym. It's a protected account. So, obviously I don't have permissions. But that means that somehow followers showed me the uid for that account which AFAIK it *shouldn't*. — John Eddy, Jul 03 '18 at 13:35
And I stand corrected, I *can* see protected accounts on other people's follow lists. Which feels odd since you can't look at a protected account and see who they follow. But regardless, that's the issue. — John Eddy, Jul 03 '18 at 22:35
I replaced my answer with a new one that addresses your recent input. — Joe Mayo, Jul 09 '18 at 03:58

c# - Listing Friends -- Hitting auth failure mid-process

1 Answers1