Scale set using keyvault in another region

Question

I'm working with an ARM template that creates a VM Scale Set for a Service Fabric cluster and associates some secrets with the VMs from a keyvault. I discovered this morning that it appears the VMs and keyvault must exist in the same region or I get an error like this:

New-AzureRmResourceGroupDeployment : 9:24:55 AM - Resource Microsoft.Compute/virtualMachineScaleSets 'StdNode' failed with message '{   "status": "Failed",   "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "KeyVaultAndVMInDifferentRegions",
        "message": "The Key Vault https://obscured.vault.azure.net/secrets/secretname/1112222aa31c4dcca4363bb0013e9999 is located in location West US, which is different from the  location of the VM, northcentralus. "
      }
    ]   } }'

This feels like an artificial limitation and is a major issue for me. I want to have a centralized keyvault where I deploy all of my secrets and utilize them from all my deployments. Having to duplicate my secrets in regions around the world seems ridiculous and VERY error prone. There should be no significant perf issue here in obtaining secrets across regions. So what is the reason behind this, and will it change?

Anyone from the Azure Scale Sets team want to offer some color to this?

score 3 · Accepted Answer · answered Sep 29 '16 at 19:30

3

the reason that we enforce region boundaries is to prevent users from creating architectures that have cross region dependencies.

For an application designed like this an outage of the japaneast datacenter will cause your VMSSes in JapanWest to not be able to successfully scale out.

Regional isolation is a key design principle of cloud based applications, and we want to prevent users from making bad choices if we can.

The reason we do not allow cross subscription references is as an important final step to prevent malicious users from using CRP as a privilege escalation mechanism to access other users secrets. There are other mechanisms which also prevent this in ARM, but are based on a configuration.

answered Sep 29 '16 at 19:30

sendmarsh

1,046
7
11

8

While I do understand your intention here it feels overly restrictive to me. Guiding architectures in what is usually the correct direction is admirable, keeping developers from designing solutions in a way you might not foresee is not a direction I'd like to see Azure go in. In my case I agree having a single point of failure via one keyvault is a bad design, but with this restriction I can't even decide to have say two or three vaults and spread their use out amongst my deployments. I am stuck having to replicate this sensitive data over all of our regional deployments. Not a fan of this. – BrettRobi Sep 29 '16 at 20:41
1

I agree with @BrettRobi, consumers should take the decision whether to have a single or region-specific KeyVaults. This restriction ultimately forces consumers to create unnecessary logic that adds complexity and potential security breaches. Why this doesn't apply to Azure Resources like storage accounts? – ksiomelo May 03 '17 at 21:05
Agree with Brett here. Would like to see restriction removed. At some point the increased complexity of tracking/managing secrets scattered across many key vaults becomes too much and devs would like to not be required to have key vault per region. – Matt Mazzola May 03 '17 at 21:06
6

But according to this link https://learn.microsoft.com/en-us/azure/key-vault/key-vault-disaster-recovery-guidance: "The contents of your key vault are replicated within the region and to a secondary region at least 150 miles away but within the same geography...In the rare event that an entire Azure region is unavailable, the requests that you make of Azure Key Vault in that region are automatically routed (failed over) to a secondary region...Again, you do not need to take any action because this happens automatically". So this seems like a pointless restriction. – Jason Boyd May 10 '17 at 05:42
Vote here - https://feedback.azure.com/forums/906355-azure-key-vault/suggestions/32184973-cert-deployment-allow-regions-to-be-different-fo – whihathac Jun 04 '18 at 21:26

score 1 · Answer 2 · answered Jan 02 '20 at 08:46

1

To overcome the problem you may simply want to apply a simple fix

Get-AzVM -ResourceGroupName "rg1" -Name "vm1" | Remove-AzVMSecret | Update-AzVM

This will remove the earlier secret and reissue a new one so that your vm is back in provisioning state.

answered Jan 02 '20 at 08:46

Shakil Bharucha

21
1
3

Thank you! this worked a treat, only thing to remember is to get into the right subscription using `Set-AzContext -SubscriptionId "xx-xx-xx-xx"` – Pete Jun 30 '20 at 23:10

score 0 · Answer 3 · answered Feb 12 '20 at 14:11

You can use an architecture of a central key vault that you access for template parameters and store those secrets in a regional key vault. Then link to the regional key vault for your scale set. If the secrets are certificates you can have an ARM function to format the certificate (as a secret) properly to be imported as a part of the OSImage property on the VM/VMSS.

A more indepth look can be found here: https://devblogs.microsoft.com/premier-developer/centralized-vm-certificate-deployment-across-multiple-regions-with-arm-templates/

Scale set using keyvault in another region

3 Answers3

Linked