IIS random hangs with Windows Azure Storage Encryption

Question

we are currently developping a web application in ASP NET MVC 4 (C#) and Angularjs where our generated content is encrypted and kept on Windows Azure.

Sometimes, randomly, IIS hangs on requests and they remain in ExecuteRequestHandler stage. Once this happened, all subsequent requests will hang and no error could be found in our IIS logs.

We use Microsoft.WindowsAzure SDK to handle everything with Windows Azure and encrypt all blobs with an encryption policy given to blob during download/upload operations (EncryptionPolicy property). We keep our encryption key in a key vault on Windows Azure also.

Here's the code to retrieve our policy:

    public async static Task<string> GetToken(string authority, string resource, string scope)
        {
            var authContext = new AuthenticationContext(authority);
            var credentials = new ClientCredential(AzureSection.ClientId, AzureSection.ClientSecret);
            var result = await authContext.AcquireTokenAsync(resource, credentials);

            if(result == null)
                throw new InvalidOperationException("Error when obtaining KeyVault token");

            return result.AccessToken;
        }

     public static BlobEncryptionPolicy GetUploadPolicy()
        {
            if (_policyUpload == null && !IsDebug)
            {
                var resolver = new KeyVaultKeyResolver(KeyVaultHelper.GetToken);

                var key = resolver.ResolveKeyAsync(AzureSection.ResourceGroup.KeyVault.Url, CancellationToken.None).GetAwaiter().GetResult();

                _policyUpload = new BlobEncryptionPolicy(key, null);
            }

            return _policyUpload;
        }

        public static BlobEncryptionPolicy GetDownloadPolicy()
        {
            if (_policyDownload == null && !IsDebug)
            {
                var resolver = new KeyVaultKeyResolver(KeyVaultHelper.GetToken);
                _policyDownload = new BlobEncryptionPolicy(null, resolver);
            }

            return _policyDownload;
        }

And how we use it during a download/upload operation:

_policyDownload = PolicyHelper.GetDownloadPolicy();



   public Task DownloadToStreamAsync(Stream output)
            {
                return !BaseBlob.Exists() ? null 
                                           : BaseBlob.DownloadToStreamAsync(output, null, new BlobRequestOptions{
                  DisableContentMD5Validation = true, 
                  EncryptionPolicy = _policyDownload 
               }, null);
            }

Where BaseBlob is a Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob and AzureSection is a section in web.config.

Our server is IIS 7.5 running on Windows Server 2008 R2. We tried on a Windows Server 2012 and IIS 8, but same issue.

All works fine when we remove EncryptionPolicy from blob operations.

Has anyone an idea why it is happening, because I couldn't find anything (since azure encryption has been implemented recently and an hanging request doesn't provide us any error) ?

EDIT

I tried to use this instead to retrieve our encryption policy (as proposed in comments of https://azure.microsoft.com/en-us/documentation/articles/storage-encrypt-decrypt-blobs-key-vault/):

using Microsoft.Azure.KeyVault.WebKey;

var kv = new KeyVaultClient(Utils.GetToken);
var key = kv.GetKeyAsync("https://contosokeyvault.vault....").Result;
var rsa = new RsaKey(key.KeyIdentifier.Identifier, key.Key.ToRSA()); 

but still same issue.

EDIT 2

I also tried to not use a static property to keep our policies for upload and download, but still same issue.

Answer 1

Know this is a bit of an old one, but we recently ran into this problem and managed to fix it. The issue is caused by a deadlock created in the async methods.

In the end a combination of suggestions made here https://stackoverflow.com/a/32429753/3391469 and here https://stackoverflow.com/a/27581462/3391469 did the trick.

blockBlob.DownloadToStream(stream, null, Task.Run(() => GetOptions()).Result);

private async static Task<BlobRequestOptions> GetOptions()
    {
        var cloudResolver = new KeyVaultKeyResolver(GetToken);
        var rsa = await cloudResolver.ResolveKeyAsync(Config.Get("AzureKeyVault"), CancellationToken.None).ConfigureAwait(false);

        var policy = new BlobEncryptionPolicy(rsa, null);
        return new BlobRequestOptions() { EncryptionPolicy = policy };
    }