4

We are using an azure storage account for our cloud services. This storage account is part of a virtual network, so access to the storage account is restricted to selected networks and the vnet is added. This works beautifully in our services.

The problem arises when we try to copy data to this storage account in an azure pipeline. Within the pipeline, we temporarily add a firewall rule to the storage account to allow traffic from the pipeline agent's Ip address to the storage account. Then we copy the data (via azcopy) and finally, remove the firewall rule. This works fine on a private agent. However, we are also using private agents hosted in azure. The problem is that if the agent runs in azure, the connection to the storage account uses private azure ip addresses, and the firewall rule doesn't work. This is specified in this doc:

Services deployed in the same region as the storage account use private Azure IP addresses for communication. Thus, you cannot restrict access to specific Azure services based on their public outbound IP address range.

Is there any way to force external routing? It seems really silly to me that with the current configuration, we are unable to connect to the storage account from within azure, and we ARE able to connect from a private agent (or any other pc) outside of azure.

I've already tried to play with the routing preference setting in the firewalls and virtual networks section of the storage account, and using the -internetrouting endpoint, but this doesn't make any difference.

Speeddymon
  • 496
  • 2
  • 20
PaulVrugt
  • 1,682
  • 2
  • 17
  • 40
  • According to your description. `The whole point is that I need to find some way to avoid using private ip addresses when contacting the storage account. This issue is not specific to a scale set. By default, any azure resource in the same region uses private IP addresses to access the storage account ` We can't do anything in Azure DevOps side. Issue should more related to Azure configuration or settings. Suggest you remove unrelated azure-pipeline tag. – PatrickLu-MSFT Nov 30 '20 at 09:03
  • 1
    I agree, I've removed the azure-pipelines tag – PaulVrugt Nov 30 '20 at 11:04

2 Answers2

1

The solution is to use private endpoints. I am dealing with the same issue and after considerable research, I found that private endpoints will facilitate access over internal IPs between the remote vnet where your agents are located and the vnet where your storage is located. I have tested this and provided details below.

After testing I found that the way this works is by creating a Private DNS in the storage vnet and setting up a vnet DNS link that allows the VMs in the remote vnet where the agents sit to resolve the storage account connection to a private IP instead of a public IP. Additionally a NIC in the remote vnet is created which provides a route to the private IPs of the storage.

So that NIC sits in the same vnet as the agents providing a route for private IP connections, and a DNS link exists to resolve storage to private IPs, and it all just works. The agents will rely on the private DNS rather than public DNS to resolve the storage hostname and so the agents will communicate with azure properly via private IPs.

Edit: I have setup a private endpoint and confirmed that it works as expected. There are some caveats to doing so with terraform which I've outlined in a related azurerm provider GitHub issue:

https://github.com/hashicorp/terraform-provider-azurerm/issues/2977#issuecomment-1011183736

Speeddymon
  • 496
  • 2
  • 20
  • I've edited the post with findings now that I've had a chance to test my theory. It works as expected with some caveats. Please read the whole post again if you had read it before because I edited the second and third paragraphs. – Speeddymon Jan 12 '22 at 21:50
  • As a follow up, I found that there are some cases where additional configuration is required. For example, if you are managing your infrastructure from a different subnet than where your endpoint is located, you may need to setup peering or an additional endpoint for the subnet where you're running Terraform from. If you're managing infrastructure across the internet, you should have a VPN and routes setup that allow you to communicate to Azure private IPs. You may need to also edit your hosts file to point to the storage account's private IP in case DNS is not available where terraform runs. – Speeddymon Feb 06 '22 at 20:11
-2

As per here, you need to allow access to the whole azure data center that match your region. So I am thinking of something automated that will query the API, retrieve the IP range and use it.

Bassam Gamal
  • 713
  • 2
  • 8
  • 24
  • Yeah there are 2 points your not taking into account. 1: Azure is using internal IP's to access the storage account when the agent is deployed in the same region, not external IP's. So whitelisting external IP's doesn't work. Secondly, we are not using microsoft hosted pipeline agents, but we have deployed a virtual machine scale set (so they are private agents running in the cloud). I'll update my question to clarify the situation – PaulVrugt Nov 27 '20 at 08:03
  • Well, as I understand now, your scale set which contains the agents is assigned to another VNET than the one which has the storage account and when you add an exception it still can't reach. Did you consider peering the two VNETs? – Bassam Gamal Nov 27 '20 at 08:50
  • Read through [here](https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-networking#public-ipv4-per-virtual-machine), maybe something you are missing? – Bassam Gamal Nov 27 '20 at 08:59
  • 1
    I don't think you're getting the point. I can't fix it by putting the agents in the same vnet (they are not even in the same subscription). And even if I could, I don't want to do that. The whole point is that I need to find some way to avoid using private ip addresses when contacting the storage account. This issue is not specific to a scale set. By default, any azure resource in the same region uses private IP addresses to access the storage account – PaulVrugt Nov 27 '20 at 09:46