What's the "right way" to use HttpClient synchronously?

Question

I used quote marks around "right way" because I'm already well aware that the right way to use an asynchronous API is to simply let the asynchronous behavior propagate throughout the entire call chain. That's not an option here.

I'm dealing with a very large and complicated system designed specifically to do batch processing synchronously in a loop.

The reason why suddenly I'm using HttpClient is because prior to now all data for the batch processing was gathered from a SQL database, and now we're adding a Web API call to the mix.

Yes, we're calling a Web API in a synchronously executing loop. I know. Rewriting the whole thing to be async just isn't an option. This is actually what we want to do. (We're minimizing the number of API calls as much as possible)

I actually did try to propagate the async behavior up the call chain, but then I found myself 50 files deep in changes, still with hundreds of compiler errors to resolve, and lost all hope. I am defeated.

So then, back to the question, given Microsoft's recommendation to never use WebRequest for new development and to instead use HttpClient, which offers only an asynchronous API, what am I to do?

Here is some pseudo-code of what I'm doing...

foreach (var thingToProcess in thingsToProcess)
{
    thingToProcess.ProcessStuff(); // This makes an API call
}

How do I implement ProcessStuff()?

My first implementation looked like this

public void ProcessStuff()
{
    var apiResponse = myHttpClient // this is an instance of HttpClient
        .GetAsync(someUrl)
        .Result;

    // do some stuff with the apiResponse
}

I was told however, that calling .Result in this manner can result in deadlocks when it's called from something like ASP.NET due to the synchronization context.

Guess what, this batch process will be kicked off from an ASP.NET controller. Yes, again, I know, this is silly. When it runs from ASP.NET it's only "batch processing" one item instead of the whole batch, but I digress, it still gets called from ASP.NET and thus I'm concerned about deadlocks.

So what's the "right way" to handle this?

Answer 1

Try the following:

var task = Task.Run(() => myHttpClient.GetAsync(someUrl)); 
task.Wait();
var response = task.Result;

Use it only when you cannot use an async method.

This method is completely deadlock free as mentioned on the MSDN blog: ASP.Net–Do not use Task .Result in main context.

Answer 2

For anyone coming across this now, .NET 5.0 has added a synchronous Send method to HttpClient. https://github.com/dotnet/runtime/pull/34948

You can therefore use this instead of SendAsync. For example

public string GetValue()
{
    var client = new HttpClient();
            
    var webRequest = new HttpRequestMessage(HttpMethod.Post, "http://your-api.com")
    {
        Content = new StringContent("{ 'some': 'value' }", Encoding.UTF8, "application/json")
    };

    var response = client.Send(webRequest);

    using var reader = new StreamReader(response.Content.ReadAsStream());
            
    return reader.ReadToEnd();
}

This code is just a simplified example, it's not production ready.

Answer 3

You could also look at using Nito.AsyncEx, which is a nuget package. I've heard of issues with using Task.Run() and this this addresses that. Here's a link to the api docs: http://dotnetapis.com/pkg/Nito.AsyncEx/4.0.1/net45/doc/Nito.AsyncEx.AsyncContext

And here's an example for using an async method in a console app: https://blog.stephencleary.com/2012/02/async-console-programs.html

Answer 4

RestSharp has an AsyncHelper that allows one to make sync calls to async methods (RestSharp in turn borrowed that class from Rebus).

I have used that class in the past (I literally just Copy&Pasted it) to make sync calls to async method and it works like charm. In case you are wondering why and how this works, there is a Blog-Post by Stephen Toub that explains how SynchronizationContext and ConfigureAwait(false) works.

To use it with the HttpClient you would do this:

AsyncHelpers.RunSync(async () => await httpClient.SendAsync(...));

If you intend to make a library/app that supports both .NET-Framework and .NET-Core you could further optimize that to this:

#if NETFRAMEWORK

//.NET-Framework does not support a sync Send
AsyncHelpers.RunSync(async () => await httpClient.SendAsync(...));

#elif NETCOREAPP

//.NET-Core does
httpClient.Send(...);

#else

//Default: Fallback to something that works on all targets.
AsyncHelpers.RunSync(async () => await httpClient.SendAsync(...));

#endif

For the sake of completeness here is AsyncHelper (again, no my implementation. I copied it from RestSharp and stripped the comments for brevity).

static class AsyncHelpers {

        public static void RunSync(Func<Task> task) {
            var currentContext = SynchronizationContext.Current;
            var customContext  = new CustomSynchronizationContext(task);

            try {
                SynchronizationContext.SetSynchronizationContext(customContext);
                customContext.Run();
            }
            finally {
                SynchronizationContext.SetSynchronizationContext(currentContext);
            }
        }
        
        public static T RunSync<T>(Func<Task<T>> task) {
            T result = default!;
            RunSync(async () => { result = await task(); });
            return result;
        }
        
        class CustomSynchronizationContext : SynchronizationContext {
            readonly ConcurrentQueue<Tuple<SendOrPostCallback, object?>> _items            = new();
            readonly AutoResetEvent                                      _workItemsWaiting = new(false);
            readonly Func<Task>                                          _task;
            ExceptionDispatchInfo?                                       _caughtException;
            bool                                                         _done;
            
            public CustomSynchronizationContext(Func<Task> task) =>
                _task = task ?? throw new ArgumentNullException(nameof(task), "Please remember to pass a Task to be executed");
            
            public override void Post(SendOrPostCallback function, object? state) {
                _items.Enqueue(Tuple.Create(function, state));
                _workItemsWaiting.Set();
            }
            
            public void Run() {
                async void PostCallback(object? _) {
                    try {
                        await _task().ConfigureAwait(false);
                    }
                    catch (Exception exception) {
                        _caughtException = ExceptionDispatchInfo.Capture(exception);
                        throw;
                    }
                    finally {
                        Post(_ => _done = true, null);
                    }
                }

                Post(PostCallback, null);

                while (!_done) {
                    if (_items.TryDequeue(out var task)) {
                        task.Item1(task.Item2);
                        if (_caughtException == null) {
                            continue;
                        }
                        _caughtException.Throw();
                    }
                    else {
                        _workItemsWaiting.WaitOne();
                    }
                }
            }
            
            public override void Send(SendOrPostCallback function, object? state) => throw new NotSupportedException("Cannot send to same thread");
            
            public override SynchronizationContext CreateCopy() => this;
        }
    }