Managed manual buffer for Task in HttpClient. MS Graph API

Question

I use GraphServiceClient to access the MS Graph API. It uses HttpClient, it was developed with an asynchronous model and should be thread-safe. I.e. there is no need to create multiple instances of GraphServiceClient for multiple requests?! I want to get authorization properties from users, I found out that if a storm is handled, then about 660 simultaneous requests are only processed. A manual buffer is needed. I can't do it, pleae help me.

If i do this:

static class POCO
{
    internal class userAAD
    {
        public IAuthenticationMethodsCollectionPage authMethods;
        public User userAttr;
    }
}

static async Task Main(string[] args)
{
    var graphClient = new GraphServiceClient(clientCertCredential, scopes);
    await FillAllAADUsers(graphClient, usersAADList);
    await FillUsersAuthMethods(graphClient, usersAADList);
}

private static async Task FillUsersAuthMethods(GraphServiceClient graphClient, List<POCO.userAAD> usersAADList)
{
    foreach (var elem in usersAADList)
    {
        await FillOneUserAuthMethods(graphClient, elem);
    }
}

private static async Task<string> FillOneUserAuthMethods(GraphServiceClient graphClient, POCO.userAAD usersAAD)
{
    while (true)
    {
        try
        {
            Console.WriteLine(usersAAD.userAttr.Id);
            var auths = await graphClient.Users[usersAAD.userAttr.Id].Authentication.Methods.Request().GetAsync();
            usersAAD.authMethods = auths;
            return usersAAD.userAttr.Id;
        }
        catch (Exception)
        {
            Console.WriteLine("+++");
            await Task.Delay(new Random().Next(1000, 4000));
        }
    }
}

The result STDOUT will of course be

ea9c9285-9e98-288e-8628-987e29eca7
9599e595-2627-2585-5205-e8956c6556
6c827ac6-a986-2772-902a-292992e6e0
58a795ee-5e99-2922-50e2-627a6e86ca
6ee29278-e678-2852-97aa-056828c567
87c207e6-2099-2e29-868e-9277ec958e
62ce69a6-2760-28e2-ac09-05c296e6a5
99567508-7927-2776-5e65-28e6c62e25
80ea76e2-5896-2c29-a86e-5e528e59c2
a25680a6-c886-250e-5ae7-6c5692c090

But if i fix the function and make a buffer, then everything does not work as expected

private static async Task FillUsersAuthMethods(GraphServiceClient graphClient, List<POCO.userAAD> usersAADList)
{
    List<Task<string>> bufferTasksList = new List<Task<string>>();

    for (int i = 0; i < usersAADList.Count; ++i)
    {
        bufferTasksList.Add(
                Task.Run(async () => {
                    var userId = await FillOneUserAuthMethods(graphClient, usersAADList[i]);
                    return userId;
                })
            );

        if (bufferTasksList.Count > 3 || i + 1 == usersAADList.Count)
        {
            await Task.WhenAll(bufferTasksList);
            bufferTasksList.Clear();
        }
    }
}

Output

53a715bb-ab66-4644-a0e4-147a1e31ca
53a715bb-ab66-4644-a0e4-147a1e31ca
53a715bb-ab66-4644-a0e4-147a1e31ca
53a715bb-ab66-4644-a0e4-147a1e31ca
66a17a08-7647-4771-ae1a-48e6c14b4a
66a17a08-7647-4771-ae1a-48e6c14b4a
66a17a08-7647-4771-ae1a-48e6c14b4a
66a17a08-7647-4771-ae1a-48e6c14b4a
cba6bb4a-68e4-4e66-68ba-6ba365444c
cba6bb4a-68e4-4e66-68ba-6ba365444c
cba6bb4a-68e4-4e66-68ba-6ba365444c
cba6bb4a-68e4-4e66-68ba-6ba365444c
b316e867-a357-465c-a64a-00e811e841
1033434a-7c7a-40ae-aa71-c311a43680
1033434a-7c7a-40ae-aa71-c311a43680
1033434a-7c7a-40ae-aa71-c311a43680

Why don't tasks work asynchronously and in parallel? why are there 4 identical and 3+1 different tasks in the output? What's going on anyway?...

Firstly you have created a closure over the Loop Variable, which is not going to work out well for you, and is one of many duplicates. — TheGeneral, Oct 13 '21 at 03:14
Secondly you are offloading an async workload with task run, which is absolutely redundant, you can just save the hot task straight to the list without task run, and is also causing your first issue by proxy — TheGeneral, Oct 13 '21 at 03:19
And lastly this whole way of limiting parallelism is suboptimal and more than little confusing `bufferTasksList.Count > 3 || i + 1 == usersAADList.Count`, there are many ways to do this, and is also very duplicated on stackoverflow — TheGeneral, Oct 13 '21 at 03:24
@TheGeneral Hello! Thank you for your attention and your help! About the variable, do you have attention `i`?! I seem to have overlooked that value-type is passed to different tasks... And what do you mean by the second remark? What does it mean "you can just save the hot task straight to the list without task run"? — KUL, Oct 13 '21 at 04:21
Without writing a full post, this should fix a lot of the problem `bufferTasksList.Add(FillOneUserAuthMethods(graphClient, usersAADList[i]))` when you call `FillOneUserAuthMethods`, it actually starts a task for you hot, this will also fix the closure issue — TheGeneral, Oct 13 '21 at 04:29
Thank you for helping me and for helping others! Everything works! I've already broken my head, but I couldn't notice the problem in the `i` variable in any way... — KUL, Oct 13 '21 at 04:38
Take a look at this https://stackoverflow.com/questions/271440/captured-variable-in-a-loop-in-c-sharp — TheGeneral, Oct 13 '21 at 04:41

KUL · Answer 1 · 2021-10-13T04:59:37.460

Parallel receipt of authentication method data from users in MS Azure via the MS Graph API

private static async Task FillUsersAuthMethods(GraphServiceClient graphClient, List<POCO.userAAD> usersAADList)
{
    List<Task<string>> bufferTasksList = new List<Task<string>>();

    for (int i = 0; i < usersAADList.Count; ++i)
    {
        bufferTasksList.Add(FillOneUserAuthMethods(graphClient, usersAADList[i]));

        if (bufferTasksList.Count > 99 || i + 1 == usersAADList.Count)
        {
            await Task.WhenAll(bufferTasksList);
            bufferTasksList.Clear();
            await Task.Delay(1000);
        }
    }
}

p.s. Search the internet for better implementations!

Managed manual buffer for Task in HttpClient. MS Graph API

1 Answers1