Azure KeyVault Active Directory AcquireTokenAsync timeout when called asynchronously

Question

I have setup Azure Keyvault on my ASP.Net MVC web application by following the example in Microsoft's Hello Key Vault sample application.

Azure KeyVault (Active Directory) AuthenticationResult by default has a one hour expiry. So after one hour, you must get a new authentication token. KeyVault is working as expected for the first hour after getting my first AuthenticationResult token, but after the 1 hour expiry, it fails to get a new token.

Unfortunately it took a failure on my production environment for me to realize this, as I never tested past one hour in development.

Anyways, after over two days of trying to figure out what was wrong with my keyvault code, I came up with a solution that fixes all of my problems - remove the asynchronous code - but it feels very hacky. I want to find out why it was not working in the first place.

My code looks like this:

public AzureEncryptionProvider() //class constructor
{
   _keyVaultClient = new KeyVaultClient(GetAccessToken);
   _keyBundle = _keyVaultClient
     .GetKeyAsync(_keyVaultUrl, _keyVaultEncryptionKeyName)
     .GetAwaiter().GetResult();
}

private static readonly string _keyVaultAuthClientId = 
    ConfigurationManager.AppSettings["KeyVaultAuthClientId"];

private static readonly string _keyVaultAuthClientSecret =
    ConfigurationManager.AppSettings["KeyVaultAuthClientSecret"];

private static readonly string _keyVaultEncryptionKeyName =
    ConfigurationManager.AppSettings["KeyVaultEncryptionKeyName"];

private static readonly string _keyVaultUrl = 
    ConfigurationManager.AppSettings["KeyVaultUrl"];

private readonly KeyBundle _keyBundle;
private readonly KeyVaultClient _keyVaultClient;

private static async Task<string> GetAccessToken(
    string authority, string resource, string scope)
{
   var clientCredential = new ClientCredential(
       _keyVaultAuthClientId, 
       _keyVaultAuthClientSecret);
   var context = new AuthenticationContext(
       authority, 
       TokenCache.DefaultShared);
   var result = context.AcquireToken(resource, clientCredential);
   return result.AccessToken;
}

The GetAccessToken method signature has to be asynchronous to pass into the new KeyVaultClient constructor, so I left the signature as async, but I removed the await keyword.

With the await keyword in there (the way it should be, and is in the sample):

private static async Task<string> GetAccessToken(string authority, string resource, string scope)
{
   var clientCredential = new ClientCredential(_keyVaultAuthClientId, _keyVaultAuthClientSecret);
   var context = new AuthenticationContext(authority, null);
   var result = await context.AcquireTokenAsync(resource, clientCredential);
   return result.AccessToken;
}

The program works fine the first time I run it. And for an hour, AcquireTokenAsync returns the same original authentication token which is great. But once the token expires, AcquiteTokenAsync should get a new token with a new expiry date. And it doesn't - the application just hangs. No error returned, nothing at all.

So calling AcquireToken instead of AcquireTokenAsync solves the problem, but I have no idea why. You'll also notice that I'm passing 'null' instead of 'TokenCache.DefaultShared' into the AuthenticationContext constructor in my sample code with async. This is to force the toke to expire immediately instead of after one hour. Otherwise, you have to wait an hour to reproduce the behavior.

I was able to reproduce this again in a brand new MVC project, so I don't think it has anything to do with my specific project. Any insight would be appreciated. But for now, I'm just not using async.

Answer 1

Problem: deadlock

Your EncryptionProvider() is calling GetAwaiter().GetResult(). This blocks the thread, and on subsequent token requests, causes a deadlock. The following code is the same as yours is but separates things to facilitate explanation.

public AzureEncryptionProvider() // runs in ThreadASP
{
    var client = new KeyVaultClient(GetAccessToken);

    var task = client.GetKeyAsync(KeyVaultUrl, KeyVaultEncryptionKeyName);

    var awaiter = task.GetAwaiter();

    // blocks ThreadASP until GetKeyAsync() completes
    var keyBundle = awaiter.GetResult();
}

In both token requests, the execution starts in the same way:

• AzureEncryptionProvider() runs in what we'll call ThreadASP.
• AzureEncryptionProvider() calls GetKeyAsync().

Then things differ. The first token request is multi-threaded:

1. GetKeyAsync() returns a Task.
2. We call GetResult() blocking ThreadASP until GetKeyAsync() completes.
3. GetKeyAsync() calls GetAccessToken() on another thread.
4. GetAccessToken() and GetKeyAsync() complete, freeing ThreadASP.
5. Our web page returns to the user. Good.

The second token request uses a single thread:

1. GetKeyAsync() calls GetAccessToken() on ThreadASP (not on a separate thread.)
2. GetKeyAsync() returns a Task.
3. We call GetResult() blocking ThreadASP until GetKeyAsync() completes.
4. GetAccessToken() must wait until ThreadASP is free, ThreadASP must wait until GetKeyAsync() completes, GetKeyAsync() must wait until GetAccessToken() completes. Uh oh.
5. Deadlock.

Why? Who knows?!?

There must be some flow control within GetKeyAsync() that relies on the state of our access token cache. The flow control decides whether to run GetAccessToken() on its own thread and at what point to return the Task.

Solution: async all the way down

To avoid a deadlock, it is a best practice "to use async all the way down." This is especially true when we are calling an async method, such as GetKeyAsync(), that is from an external library. It is important not force the method to by synchronous with Wait(), Result, or GetResult(). Instead, use async and await because await pauses the method instead of blocking the whole thread.

Async controller action

public class HomeController : Controller
{
    public async Task<ActionResult> Index()
    {
        var provider = new EncryptionProvider();
        await provider.GetKeyBundle();
        var x = provider.MyKeyBundle;
        return View();
    }
}

Async public method

Since a constructor cannot be async (because async methods must return a Task), we can put the async stuff into a separate public method.

public class EncryptionProvider
{
    //
    // authentication properties omitted

    public KeyBundle MyKeyBundle;

    public EncryptionProvider() { }

    public async Task GetKeyBundle()
    {
        var keyVaultClient = new KeyVaultClient(GetAccessToken);
        var keyBundleTask = await keyVaultClient
            .GetKeyAsync(KeyVaultUrl, KeyVaultEncryptionKeyName);
        MyKeyBundle = keyBundleTask;
    }

    private async Task<string> GetAccessToken(
        string authority, string resource, string scope)
    {
        TokenCache.DefaultShared.Clear(); // reproduce issue 
        var authContext = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCredential = new ClientCredential(ClientIdWeb, ClientSecretWeb);
        var result = await authContext.AcquireTokenAsync(resource, clientCredential);
        var token = result.AccessToken;
        return token;
    }
}

Mystery solved. :) Here is a final reference that helped my understanding.

Console App

My original answer had this console app. It worked as an initial troubleshooting step. It did not reproduce the problem.

The console app loops every five minutes, repeatedly asking for a new access token. At each loop, it outputs the current time, the expiry time, and the name of the retrieved key.

On my machine, the console app ran for 1.5 hours and successfully retrieved a key after expiration of the original.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Azure.KeyVault;
using Microsoft.IdentityModel.Clients.ActiveDirectory;

namespace ConsoleApp
{
    class Program
    {
        private static async Task RunSample()
        {
            var keyVaultClient = new KeyVaultClient(GetAccessToken);

            // create a key :)
            var keyCreate = await keyVaultClient.CreateKeyAsync(
                vault: _keyVaultUrl,
                keyName: _keyVaultEncryptionKeyName,
                keyType: _keyType,
                keyAttributes: new KeyAttributes()
                {
                    Enabled = true,
                    Expires = UnixEpoch.FromUnixTime(int.MaxValue),
                    NotBefore = UnixEpoch.FromUnixTime(0),
                },
                tags: new Dictionary<string, string> {
                    { "purpose", "StackOverflow Demo" }
                });

            Console.WriteLine(string.Format(
                "Created {0} ",
                keyCreate.KeyIdentifier.Name));

            // retrieve the key
            var keyRetrieve = await keyVaultClient.GetKeyAsync(
                _keyVaultUrl,
                _keyVaultEncryptionKeyName);

            Console.WriteLine(string.Format(
                "Retrieved {0} ",
                keyRetrieve.KeyIdentifier.Name));
        }

        private static async Task<string> GetAccessToken(
            string authority, string resource, string scope)
        {
            var clientCredential = new ClientCredential(
                _keyVaultAuthClientId,
                _keyVaultAuthClientSecret);

            var context = new AuthenticationContext(
                authority,
                TokenCache.DefaultShared);

            var result = await context.AcquireTokenAsync(resource, clientCredential);

            _expiresOn = result.ExpiresOn.DateTime;

            Console.WriteLine(DateTime.UtcNow.ToShortTimeString());
            Console.WriteLine(_expiresOn.ToShortTimeString());

            return result.AccessToken;
        }

        private static DateTime _expiresOn;
        private static string
            _keyVaultAuthClientId = "xxxxx-xxx-xxxxx-xxx-xxxxx",
            _keyVaultAuthClientSecret = "xxxxx-xxx-xxxxx-xxx-xxxxx",
            _keyVaultEncryptionKeyName = "MYENCRYPTIONKEY",
            _keyVaultUrl = "https://xxxxx.vault.azure.net/",
            _keyType = "RSA";

        static void Main(string[] args)
        {
            var keepGoing = true;
            while (keepGoing)
            {
                RunSample().GetAwaiter().GetResult();
                // sleep for five minutes
                System.Threading.Thread.Sleep(new TimeSpan(0, 5, 0)); 
                if (DateTime.UtcNow > _expiresOn)
                {
                    Console.WriteLine("---Expired---");
                    Console.ReadLine();
                }
            }
        }
    }
}

Answer 2

I have the same challenge you have. I am assuming that you've also seen the sample published at https://azure.microsoft.com/en-us/documentation/articles/key-vault-use-from-web-application/

There is a big difference between what that sample does and what my code does (and I think the intent of your code is). In the sample, they retrieve a secrete and store it in the web application as a static member of their Utils class. Thus, the sample retrieves a secret one time for the entire run time of the application.

In my case, I am retrieving a different key for different purposes at different times during the application's run time.

Additionally, the sample download you linked to uses an X.509 certificate to authenticate the web application to KeyVault, rather than client secret. It's possible there is an issue with that too.

I saw the chat with @shaun-luttin concluding you caused a deadlock, but that's not the whole story I think. I don't use .GetAwaiter().GetResult() or call an async method from a ctor.