0

I am setting up separate .NET (C#) tasks to run in parallel in an ASP.NET project (.NET 4.6.2) and want to limit the number of them that can run at once to 10. Most examples I have seen to accomplish this make use of the SemaphoreSlim and I have set up my logic the same as most of those examples:

            // Limit the concurrent number of threads to 10
            var throttler = new SemaphoreSlim(10);

            // Create a list of tasks to run
            var tasks = images.Select(async image =>
            {
                _logger.Info("WAITING FOR THROTTLER");
                await throttler.WaitAsync();
                try
                {
                    _logger.Info("ABOUT TO DOWNLOAD IMAGE FROM API A");
                    var imageData = await DownloadImage(...);

                    if (imageData != null)
                    {
                        _logger.Info("ABOUT TO UPLOAD IMAGE TO API B");
                        await AddFileAsync(...);
                    }
                }
                catch (Exception e)
                {
                    _logger.Error("Failed on UploadImages: " + e.Message);
                }
                finally
                {
                    throttler.Release();// Always release the semaphore when done
                }
            });

            await Task.WhenAll(tasks);// Now we actually run the tasks

For context, this code runs as fire-and-forget and is triggered as part of a web request. Each task downloads image data from an API (A) and then uploads that data to a different API (B) via HttpClient.

Testing this with 12 images, sometimes it works fine. But, there is an intermittent issue where sometimes 2 images never make it through, no error is thrown or returned from the API calls. I placed the logs in to narrow it down and found that after the first 10 process, the last 2 just never get passed the log: "WAITING FOR THROTTLER" so it seems to be stuck on waiting for the semaphore to release?

Output logs from one of these issue instances:

2023-03-15 18:33:17,049 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,049 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,050 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,050 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,051 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,051 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,053 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,053 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,054 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,054 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,055 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,055 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,056 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,056 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,057 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,058 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,059 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,059 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
2023-03-15 18:33:17,059 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,059 [41] INFO MyService  ABOUT TO DOWNLOAD IMAGE FROM API A
SEE HERE TWO LINES BELOW
2023-03-15 18:33:17,061 [41] INFO MyService  WAITING FOR THROTTLER
2023-03-15 18:33:17,061 [41] INFO MyService  WAITING FOR THROTTLER
SEE HERE TWO LINES ABOVE
2023-03-15 18:33:17,996 [74] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,231 [40] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,241 [40] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,247 [51] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,253 [51] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,259 [51] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,265 [79] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,271 [79] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,277 [40] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B
2023-03-15 18:33:18,309 [74] INFO MyService  ABOUT TO UPLOAD IMAGE TO API B

I have outlined the two lines in the logs that suggest to me the issue is related to the semaphore. There are 10 logs for calling "DOWNLOAD" and "UPLOAD", but all 12 show "WAITING FOR THROTTLER" So you can see that two of the tasks never reach the point of calling the API and are stuck waiting on the semaphore.

I wonder why this happens though especially when it works sometimes, seems to be some sort of timing issue?

*I'll also note that using Parallel.ForEachAsync is not an option as this project is running on .NET 4.6.2 and upgrading is not an option right now.

Reddy
  • 481
  • 1
  • 7
  • 20
  • If you comment out the `await throttler.WaitAsync();` and `throttler.Release();` lines, do all tasks complete as expected? I am asking in order to make sure that the issue is related with the semaphore, and it's not something else. – Theodor Zoulias Mar 16 '23 at 01:14
  • What is the type of your project? Win Forms? ASP.NET? Console app? – Theodor Zoulias Mar 16 '23 at 01:16
  • @TheodorZoulias I would think by my findings with the logs that it has proven to be related to the semaphore as the 2 issue tasks never reach the log for hitting the API. I can give it a try though. And Project is ASP.NET using .NET 4.6.2. – Reddy Mar 16 '23 at 01:23
  • Yeah confirmed it works fine when I comment out the lines you mentioned. – Reddy Mar 16 '23 at 01:38
  • 1
    Inside the `​DownloadImage` and `AddFileAsync` methods, do you have any `.Result` or `.Wait()` or `.GetAwaiter().GetResult()` on any non-complete task? – Theodor Zoulias Mar 16 '23 at 01:50
  • 1
    Does this code run as part of a web request, or it's [fire-and-forget](https://blog.stephencleary.com/2014/06/fire-and-forget-on-asp-net.html)? – Theodor Zoulias Mar 16 '23 at 01:55
  • 1
    DownloadImage is an async method with return type Task. It awaits on a call to HttpClient.GetAsync(url). It returns null if the request does not return 200 OK. AddFileAsync is an async method that is of type Task so it does not return anything. It awaits on a call to HttpClient.SendAsync(requestMessage) This logic is running as a fire-and-forget, but it is triggered as part of a web request. – Reddy Mar 16 '23 at 02:06
  • You could consider editing the question and adding the information about the project type (ASP.NET), and the fire-and-forget nature of the code. I think this info is important. Personally I am not seeing anything wrong with the way you use the semaphore, and I would be surprised if eventually the semaphore is proven to be directly related to the problem. – Theodor Zoulias Mar 16 '23 at 02:30
  • 1
    I have edited it to include the information you mentioned. I also included the output logs that suggest to me the issue being related to the semaphore. – Reddy Mar 16 '23 at 02:51
  • Could you try initializing the semaphore with a smaller `initialCount`, like `2`, and see what happens? Btw it's a good idea to set the `maxCount` too, with the same value as the `initialCount`. For example `new SemaphoreSlim(2, 2);`. Setting the second parameter shouldn't have any noticeable effect. It's just a preventive measure against misuse. – Theodor Zoulias Mar 16 '23 at 03:03
  • 1
    Maybe add logging after the release to ensure it is actually releasing the semaphore. – AndrewS Mar 16 '23 at 03:11
  • @TheodorZoulias I applied your suggestion and again sometimes all of them process fine, but then in the instances that replicate the issue, usually about 4 go through, and 8 never process. I added more logging and it seems to have narrowed down the issue to the Upload call to API B. (Not the semaphore like you said) It seems that on the second batch of calls, it never returns from this line: `var httpResponseMessage = await _httpClient.SendAsync(requestMessage)`, is there something about HttpClient that causes it to lock with concurrent calls coming in..? – Reddy Mar 16 '23 at 14:51
  • Note that I confirmed that these hanging requests do not make it to the endpoint, so it seems to be getting hung up in the HttpClient somewhere. – Reddy Mar 16 '23 at 15:01
  • 1
    Check out these two questions, they might be relevant: [What is HttpClient's default maximum connections](https://stackoverflow.com/questions/31735569/what-is-httpclients-default-maximum-connections) and [How can I programmatically remove the 2 connection limit in WebClient](https://stackoverflow.com/questions/866350/how-can-i-programmatically-remove-the-2-connection-limit-in-webclient). I would also suggest to test your code as a Console application, to make sure that the problem is not caused by the host terminating the process before the fire-and-forget task completes. – Theodor Zoulias Mar 16 '23 at 15:07
  • I found a number of suggestions on using `.ConfigureAwait(false)`: https://stackoverflow.com/questions/10343632/httpclient-getasync-never-returns-when-using-await-async I added this to the Http call and it made it passed that point, but then it got stuck just after it, so I added the same thing to the AddFileAsync call, then it made it further but got stuck after releasing the semaphore. Finally added it to the `throttler.WaitAsync()` as well and now it seems to be working, all of the images are consistently being uploaded. Will do some more testing to confirm. – Reddy Mar 16 '23 at 17:30
  • This indicates that you call `.Result` or `.Wait()` or `.GetAwaiter().GetResult()` somewhere. Search your project for these keywords. I am pretty sure that you'll find at least one. – Theodor Zoulias Mar 16 '23 at 17:40

1 Answers1

2

this code runs as fire-and-forget

Found the problem. ^

Fire-and-forget code is unreliable in the general case.

no error is thrown or returned from the API calls.

Also normal for fire-and-forget code.

OK, so here's the problem(s) with fire-and-forget:

  • ASP.NET is aware of requests and responses. Anything outside of that (i.e., fire-and-forget) is code that exists outside of the knowledge of ASP.NET (by default).
  • Therefore, whenever ASP.NET exits (i.e., whenever you upgrade your code), this request-extrinsic work will just stop.
  • There's no logging AFAIK. There might be an exception; if so, it will be ignored.
  • There's no error code returned from API calls, of course, because this is request-extrinsic code so there is no response to hold the error code.

These are the normal problems with fire-and-forget code. If you want to ensure the work gets done, build a proper distributed architecture. (Links are to my blog). It's some work, but that's what is necessary for reliable processing.

In addition to the normal problems above, your fire-and-forget code has an additional problem:

I found a number of suggestions on using .ConfigureAwait(false)

What's likely happening is that your request code is just calling it like _ = MyFireAndForgetMethod(...);. The problem is that ASP.NET pre-Core has a request context, and await will capture that context and return on it (by default; the ConfigureAwait(false) overrides this behavior and tells it not to return on the request context). This is a problem because after the request is completed (i.e., after the response has been sent), that request context is no longer valid. Code often (but not always) fails when attempting to use a request context that is no longer valid. (Link is to my blog)

To avoid this more completely, you can wrap your fire-and-forget in Task.Run: _ = Task.Run(() => MyFireAndForgetMethod(...));. Task.Run is generally considered an antipattern on ASP.NET, but you already have a way bigger antipattern with the fire-and-forget code, so adding a smaller antipattern to avoid this problem with the bigger antipattern is not a huge deal.

Of course, you still have all the normal (and unavoidable) inherent problems with fire-and-forget to deal with. The only solution I could actually recommend is a distributed architecture.

Note that this had nothing to do with tasks or concurrency or SemaphoreSlim; the problem is the result of fire-and-forget.

Stephen Cleary
  • 437,863
  • 77
  • 675
  • 810
  • So is it possible that the operation hangs because the ASP.NET synchronization context just dies, and silently stops executing the scheduled continuations? – Theodor Zoulias Mar 17 '23 at 13:26
  • 1
    @TheodorZoulias: I have observed that behavior, yes. There's some kind of cleanup that happens some time after the request completes after which continuations are not run. I'm not sure whether it actually silently drops continuations or if there's an exception that is silently swallowed. I also don't know if request contexts are reused, which would make this behavior more rare on more busy server. – Stephen Cleary Mar 17 '23 at 16:58
  • Thank you for the detailed response. I'm not going to pretend I can completely wrap my head around all of that, but it seems this issue is more complex than I initially realized. This upload process is dealing with fairly large files and takes quite some time so we changed it to fire-and-forget so as not to block the user and the upload can continue in the background, but it's good to know there are ramifications with such an approach. – Reddy Mar 21 '23 at 18:52