Terraform - How to attach SSL certificate stored in Azure KeyVault to an Application Gateway

Question

I have a Terraform script that create an Azure Key Vault, imports my SSL certificate (3DES .pfx file with a password), and creates an Application Gateway with a HTTP listener. I'm trying to change this to a HTTPS listener that uses my SSL certificate from KeyVault.

I've stepped through this process manually in Azure Portal and I have this working with PowerShell. Unfortunately I don't find Terraform's documentation clear on how this is supposed to be achieved.

Here are relevant snippets of my Application Gateway and certificate resources:

resource "azurerm_application_gateway" "appgw" {
  name                = "my-appgw"
  location            = "australiaeast"
  resource_group_name = "my-rg"
  
  http_listener {
    protocol                       = "https"
    ssl_certificate_name           = "appgw-listener-cert"
    ...
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.appgw_uaid.id]
  }

  ssl_certificate {
    key_vault_secret_id = azurerm_key_vault_certificate.ssl_cert.secret_id
    name                = "appgw-listener-cert"
  }

  ...
}

resource "azurerm_key_vault" "kv" {
  name                       = "my-kv"
  location                   = "australiaeast"
  resource_group_name        = "my-rg"
  ...
  access_policy {
    object_id    = data.azurerm_client_config.current.object_id
    tenant_id    = data.azurerm_client_config.current.tenant_id

    certificate_permissions = [
      "Create",
      "Delete",
      "DeleteIssuers",
      "Get",
      "GetIssuers",
      "Import",
      "List",
      "ListIssuers",
      "ManageContacts",
      "ManageIssuers",
      "Purge",
      "SetIssuers",
      "Update"
    ]

    key_permissions = [
      "Backup",
      "Create",
      "Decrypt",
      "Delete",
      "Encrypt",
      "Get",
      "Import",
      "List",
      "Purge",
      "Recover",
      "Restore",
      "Sign",
      "UnwrapKey",
      "Update",
      "Verify",
      "WrapKey"
    ]

    secret_permissions = [
      "Backup",
      "Delete",
      "Get",
      "List",
      "Purge",
      "Restore",
      "Restore",
      "Set"
    ]
  }

  access_policy {
    object_id    = azurerm_user_assigned_identity.uaid_appgw.principal_id
    tenant_id    = data.azurerm_client_config.current.tenant_id

    secret_permissions = [
      "Get"
    ]
  }
}

resource "azurerm_key_vault_certificate" "ssl_cert" {
  name         = "my-ssl-cert"
  key_vault_id = azurerm_key_vault.kv.id

  certificate {
    # These are stored as sensitive variables in Terraform Cloud
    # ssl_cert_b64 value was retrieved by: $ cat my-ssl-cert.pfx | base64 > o.txt
    contents = var.ssl_cert_b64
    password = var.ssl_cert_passwd
  }

  certificate_policy {
    issuer_parameters {
      name = "Unknown"
    }

    key_properties {
      exportable = false
      key_size   = 2048
      key_type   = "RSA"
      reuse_key  = false
    }

    secret_properties {
      content_type = "application/x-pkcs12"
    }
  }
}

Here is the (sanitised) error I get in Terraform Cloud:

Error: waiting for create/update of Application Gateway: (Name "my-appgw" / Resource Group "my-rg"): Code="ApplicationGatewayKeyVaultSecretException" Message="Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/1324/resourceGroups/my-rg/providers/Microsoft.Network/applicationGateways/my-appgw'. See details below:" Details=[{"code":"ApplicationGatewaySslCertificateDoesNotHavePrivateKey","message":"Certificate /subscriptions/1324/resourceGroups/my-rg/providers/Microsoft.Network/applicationGateways/my-appgw/sslCertificates/appgw-listener-cert does not have Private Key."}]

I downloaded the certificate from Key Vault and it appears to be a valid, not corrupted or otherwise broken. I don't understand why the error says it doesn't have a Private Key.

Can someone point out what I've missed or I'm doing wrong?

score 11 · Accepted Answer · answered Sep 17 '21 at 09:49

I tested 2 scenarios in my environment :

Scenario 1: Generating a new certificate in Keyvault and uploading it in application gateway ssl certificate.

provider "azurerm" {
    features{}
}
data "azurerm_client_config" "current" {}

data "azurerm_resource_group" "example"{
    name = "ansumantest"
}

resource "azurerm_user_assigned_identity" "base" {
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  name                = "mi-appgw-keyvault"
}


resource "azurerm_key_vault" "kv" {
  name                       = "ansumankeyvault01"
  location                   = data.azurerm_resource_group.example.location
  resource_group_name        = data.azurerm_resource_group.example.name
  tenant_id = data.azurerm_client_config.current.tenant_id
  sku_name = "standard"
  access_policy {
    object_id    = data.azurerm_client_config.current.object_id
    tenant_id    = data.azurerm_client_config.current.tenant_id

    certificate_permissions = [
      "Create",
      "Delete",
      "DeleteIssuers",
      "Get",
      "GetIssuers",
      "Import",
      "List",
      "ListIssuers",
      "ManageContacts",
      "ManageIssuers",
      "Purge",
      "SetIssuers",
      "Update"
    ]

    key_permissions = [
      "Backup",
      "Create",
      "Decrypt",
      "Delete",
      "Encrypt",
      "Get",
      "Import",
      "List",
      "Purge",
      "Recover",
      "Restore",
      "Sign",
      "UnwrapKey",
      "Update",
      "Verify",
      "WrapKey"
    ]

    secret_permissions = [
      "Backup",
      "Delete",
      "Get",
      "List",
      "Purge",
      "Restore",
      "Restore",
      "Set"
    ]
  }

  access_policy {
    object_id    = azurerm_user_assigned_identity.base.principal_id
    tenant_id    = data.azurerm_client_config.current.tenant_id

    secret_permissions = [
      "Get"
    ]
  }
}

output "secret_identifier" {
  value = azurerm_key_vault_certificate.example.secret_id
}

resource "azurerm_key_vault_certificate" "example" {
  name         = "generated-cert"
  key_vault_id = azurerm_key_vault.kv.id

  certificate_policy {
    issuer_parameters {
      name = "Self"
    }

    key_properties {
      exportable = true
      key_size   = 2048
      key_type   = "RSA"
      reuse_key  = true
    }

    lifetime_action {
      action {
        action_type = "AutoRenew"
      }

      trigger {
        days_before_expiry = 30
      }
    }

    secret_properties {
      content_type = "application/x-pkcs12"
    }

    x509_certificate_properties {
      # Server Authentication = 1.3.6.1.5.5.7.3.1
      # Client Authentication = 1.3.6.1.5.5.7.3.2
      extended_key_usage = ["1.3.6.1.5.5.7.3.1"]

      key_usage = [
        "cRLSign",
        "dataEncipherment",
        "digitalSignature",
        "keyAgreement",
        "keyCertSign",
        "keyEncipherment",
      ]

      subject_alternative_names {
        dns_names = ["internal.contoso.com", "domain.hello.world"]
      }

      subject            = "CN=hello-world"
      validity_in_months = 12
    }
  }
}

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  address_space       = ["10.254.0.0/16"]
}

resource "azurerm_subnet" "frontend" {
  name                 = "frontend"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.254.0.0/24"]
}

resource "azurerm_subnet" "backend" {
  name                 = "backend"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.254.2.0/24"]
}

resource "azurerm_public_ip" "example" {
  name                = "example-pip"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  allocation_method   = "Static"
  sku = "standard"
}

#&nbsp;since these variables are re-used - a locals block makes this more maintainable
locals {
  backend_address_pool_name      = "${azurerm_virtual_network.example.name}-beap"
  frontend_port_name             = "${azurerm_virtual_network.example.name}-feport"
  frontend_ip_configuration_name = "${azurerm_virtual_network.example.name}-feip"
  http_setting_name              = "${azurerm_virtual_network.example.name}-be-htst"
  listener_name                  = "${azurerm_virtual_network.example.name}-httplstn"
  request_routing_rule_name      = "${azurerm_virtual_network.example.name}-rqrt"
  redirect_configuration_name    = "${azurerm_virtual_network.example.name}-rdrcfg"

}

resource "null_resource" "previous" {}

resource "time_sleep" "wait_240_seconds" {
  depends_on = [azurerm_key_vault.kv]

  create_duration = "240s"
}

resource "azurerm_application_gateway" "network" {
  name                = "example-appgateway"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location

  sku {
    name     = "Standard_v2"
    tier     = "Standard_v2"
    capacity = 2
  }

  gateway_ip_configuration {
    name      = "my-gateway-ip-configuration"
    subnet_id = azurerm_subnet.frontend.id
  }

  frontend_port {
    name = local.frontend_port_name
    port = 443
  }

  frontend_ip_configuration {
    name                 = local.frontend_ip_configuration_name
    public_ip_address_id = azurerm_public_ip.example.id
  }

  backend_address_pool {
    name = local.backend_address_pool_name
  }

  backend_http_settings {
    name                  = local.http_setting_name
    cookie_based_affinity = "Disabled"
    path                  = "/path1/"
    port                  = 443
    protocol              = "Https"
    request_timeout       = 60
  }

  http_listener {
    name                           = local.listener_name
    frontend_ip_configuration_name = local.frontend_ip_configuration_name
    frontend_port_name             = local.frontend_port_name
    protocol                       = "Https"
    ssl_certificate_name = "app_listener"
  }

  identity {
    type = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.base.id]
  }

  ssl_certificate {
    name = "app_listener"
    key_vault_secret_id = azurerm_key_vault_certificate.example.secret_id
  }

  request_routing_rule {
    name                       = local.request_routing_rule_name
    rule_type                  = "Basic"
    http_listener_name         = local.listener_name
    backend_address_pool_name  = local.backend_address_pool_name
    backend_http_settings_name = local.http_setting_name
  }
  depends_on = [time_sleep.wait_240_seconds]
}

Output:

Scenario 2 : Using one certificate which I import from local machine to keyvault and using it in application gateway.

provider "azurerm" {
    features{}
}
data "azurerm_client_config" "current" {}

data "azurerm_resource_group" "example"{
    name = "ansumantest"
}

resource "azurerm_user_assigned_identity" "base" {
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  name                = "mi-appgw-keyvault"
}


resource "azurerm_key_vault" "kv" {
  name                       = "ansumankeyvault01"
  location                   = data.azurerm_resource_group.example.location
  resource_group_name        = data.azurerm_resource_group.example.name
  tenant_id = data.azurerm_client_config.current.tenant_id
  sku_name = "standard"
  access_policy {
    object_id    = data.azurerm_client_config.current.object_id
    tenant_id    = data.azurerm_client_config.current.tenant_id

    certificate_permissions = [
      "Create",
      "Delete",
      "DeleteIssuers",
      "Get",
      "GetIssuers",
      "Import",
      "List",
      "ListIssuers",
      "ManageContacts",
      "ManageIssuers",
      "Purge",
      "SetIssuers",
      "Update"
    ]

    key_permissions = [
      "Backup",
      "Create",
      "Decrypt",
      "Delete",
      "Encrypt",
      "Get",
      "Import",
      "List",
      "Purge",
      "Recover",
      "Restore",
      "Sign",
      "UnwrapKey",
      "Update",
      "Verify",
      "WrapKey"
    ]

    secret_permissions = [
      "Backup",
      "Delete",
      "Get",
      "List",
      "Purge",
      "Restore",
      "Restore",
      "Set"
    ]
  }

  access_policy {
    object_id    = azurerm_user_assigned_identity.base.principal_id
    tenant_id    = data.azurerm_client_config.current.tenant_id

    secret_permissions = [
      "Get"
    ]
  }
}

output "secret_identifier" {
  value = azurerm_key_vault_certificate.example.secret_id
}

resource "azurerm_key_vault_certificate" "example" {
  name         = "imported-cert"
  key_vault_id = azurerm_key_vault.kv.id

  certificate {
    contents = filebase64("C:/appgwlistner.pfx")
    password = "password"
  }

  certificate_policy {
    issuer_parameters {
      name = "Self"
    }

    key_properties {
      exportable = true
      key_size   = 2048
      key_type   = "RSA"
      reuse_key  = false
    }

    secret_properties {
      content_type = "application/x-pkcs12"
    }
  }
}

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  address_space       = ["10.254.0.0/16"]
}

resource "azurerm_subnet" "frontend" {
  name                 = "frontend"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.254.0.0/24"]
}

resource "azurerm_subnet" "backend" {
  name                 = "backend"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.254.2.0/24"]
}

resource "azurerm_public_ip" "example" {
  name                = "example-pip"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  allocation_method   = "Static"
  sku = "standard"
}

#&nbsp;since these variables are re-used - a locals block makes this more maintainable
locals {
  backend_address_pool_name      = "${azurerm_virtual_network.example.name}-beap"
  frontend_port_name             = "${azurerm_virtual_network.example.name}-feport"
  frontend_ip_configuration_name = "${azurerm_virtual_network.example.name}-feip"
  http_setting_name              = "${azurerm_virtual_network.example.name}-be-htst"
  listener_name                  = "${azurerm_virtual_network.example.name}-httplstn"
  request_routing_rule_name      = "${azurerm_virtual_network.example.name}-rqrt"
  redirect_configuration_name    = "${azurerm_virtual_network.example.name}-rdrcfg"

}

resource "null_resource" "previous" {}

resource "time_sleep" "wait_240_seconds" {
  depends_on = [azurerm_key_vault.kv]

  create_duration = "240s"
}

resource "azurerm_application_gateway" "network" {
  name                = "example-appgateway"
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location

  sku {
    name     = "Standard_v2"
    tier     = "Standard_v2"
    capacity = 2
  }

  gateway_ip_configuration {
    name      = "my-gateway-ip-configuration"
    subnet_id = azurerm_subnet.frontend.id
  }

  frontend_port {
    name = local.frontend_port_name
    port = 443
  }

  frontend_ip_configuration {
    name                 = local.frontend_ip_configuration_name
    public_ip_address_id = azurerm_public_ip.example.id
  }

  backend_address_pool {
    name = local.backend_address_pool_name
  }

  backend_http_settings {
    name                  = local.http_setting_name
    cookie_based_affinity = "Disabled"
    path                  = "/path1/"
    port                  = 443
    protocol              = "Https"
    request_timeout       = 60
  }

  http_listener {
    name                           = local.listener_name
    frontend_ip_configuration_name = local.frontend_ip_configuration_name
    frontend_port_name             = local.frontend_port_name
    protocol                       = "Https"
    ssl_certificate_name = "app_listener"
  }

  identity {
    type = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.base.id]
  }

  ssl_certificate {
    name = "app_listener"
    key_vault_secret_id = azurerm_key_vault_certificate.example.secret_id
  }

  request_routing_rule {
    name                       = local.request_routing_rule_name
    rule_type                  = "Basic"
    http_listener_name         = local.listener_name
    backend_address_pool_name  = local.backend_address_pool_name
    backend_http_settings_name = local.http_setting_name
  }
  depends_on = [time_sleep.wait_240_seconds]
}

Outputs:

Note:

Please make sure to have the pfx certificate with private keys. While you are exporting a pfx certificate using a security certificate, please make sure to have the following propeties selected as shown below and then give a password and export it.

Again, this is a very thorough reply so thanks. I don't think it's an accurate test, though, because I mentioned I'm using Terraform Cloud which means it's not possible to import the SSL cert from local disk as you've done. I notice, however, that you set "exportable" to true while I set it to false. I changed my script so "exportable" is true and the errors were resolved! What exactly does "exportable" do? It's not documented very well by TF. — wertyq, Sep 23 '21 at 12:09
@wertyq, Glad to be of help! exportable should be set to true as you have to get the certificate from keyvault to be used by any other azure resources. if its set to false then the service which will refer the keyvault for certificate will be able to see it but not able to use it. — Ansuman Bal, Sep 23 '21 at 12:21
Thanks for clarifying, this is what I suspected. I initially thought "exportable" meant it could be exported by a user from Azure to elsewhere like a local disk. — wertyq, Sep 23 '21 at 12:23

score 1 · Answer 2 · answered Sep 16 '21 at 10:10

The issue is that there isn't any access policy defined for the app gateway in the keyvault for which it not able to get the certififcate.

So, inorder to resolve this , you have to add an acess policy for the managed identity that is being used by the application gateway. So, after creating managed identity and before using in application gateway, you have to use something like below:

provider "azurerm" {
    features{}
}
data "azurerm_client_config" "current" {}

resource "azurerm_user_assigned_identity" "base" {
  resource_group_name = "yourresourcegroup"
  location            = "resourcegrouplocation"
  name                = "mi-appgw-keyvault"
}

data "azurerm_key_vault" "example"{
    name = "testansumankeyvault-01"
    resource_group_name = "yourresourcegroup"
} 
resource "azurerm_key_vault_access_policy" "example" {
  key_vault_id = data.azurerm_key_vault.example.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.base.principal_id

  key_permissions = [
    "Get",
  ]

  certificate_permissions = [
      "Get",
  ]

  secret_permissions = [
    "Get",
  ]
}

So , Only after the above is done you can use something similar to below as per your requirement:

data "azurerm_user_assigned_identity" "example" {
  name                = "mi-appgw-keyvault"
  resource_group_name = "yourresourcegroup"
}
data "azurerm_key_vault" "example"{
    name = "testansumankeyvault-01"
    resource_group_name = "yourresourcegroup"
} 
resource "azurerm_application_gateway" "appgw" {
  name                = "my-appgw"
  location            = "australiaeast"
  resource_group_name = "my-rg"
  
  http_listener {
    protocol                       = "https"
    ssl_certificate_name           = "appgw-listener-cert"
    ...
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [data.azurerm_user_assigned_identity.example.id]
  }

  ssl_certificate {
    key_vault_secret_id = azurerm_key_vault_certificate.ssl_cert.secret_id
    name                = "appgw-listener-cert"
  }

  ...
}
    
data "azurerm_key_vault_certificate" "example" {
  name         = "secret-sauce"
  key_vault_id = data.azurerm_key_vault.example.id
}

Note:

I have used exisitng keyvault to set the keyvault access policy for testing and also a exisitng certificate in keyvault. If you are creating new then please use 2 deployments:

Deploy Keyvault,managed_identity ,access policy and certificate for keyvault first.
Then use data sources for keyvault, managed identity and certificate and then deploy the application gateway with ssl certificate referencing from keyvault.

I appreciate the detailed reply however I have made a mistake - I do have an inline access policy and it's working. My mistake for not including it, I'll update my question. — wertyq, Sep 16 '21 at 11:34
and also creating the keyvault and adding access policy and everything at the same time would error out the same as it will take few mins to apply the access policy for managed identity.. and then again it will fail while accessing the keyvault — Ansuman Bal, Sep 16 '21 at 11:47
"and also creating the keyvault and adding access policy and everything at the same time would error out the same as it will take few mins to apply the access policy for managed identity.. and then again it will fail while accessing the keyvault" - both the inline access_policy blocks work just fine, I have a 60s time_sleep on the application gateway resource. I was previously using azurerm_key_vault_access_policy resources and they did not work at all. None of this really addresses the error message, though, which is Terraform/Azure saying the cert doesn't have a private key when it does. — wertyq, Sep 17 '21 at 02:33
Hello @wertyq, seeing the edit you have made now seems you have added the second access policy as well and still getting the error . Let me test it in your way and provide an solution. — Ansuman Bal, Sep 17 '21 at 03:11
@wertyq, may I know if you are able to add the same cert from keyvault to a application gateway from portal? — Ansuman Bal, Sep 17 '21 at 04:42
@wertyq, lets move the discussion to chat : https://chat.stackoverflow.com/rooms/237199/room-for-wertyq-and-ansuman . — Ansuman Bal, Sep 17 '21 at 06:30

Terraform - How to attach SSL certificate stored in Azure KeyVault to an Application Gateway

2 Answers2

Linked