Azure Devops - Get All variables with specific prefix dynamically (azure cli)

Question

Currently I'm using azure devops to deploy an app container to azure.

I've created my release pipeline and added each ENVIRONMENT VARIABLE manually through the interface.

I wondering if is there any way to get it dynamically, I mean add the environment variables with a given prefix and add to the command.

Currently this is the command that I'm using:

az containerapp create --resource-group $(ResourceGroupName) --name $(ContainerAppName) --container-name $(ContainerAppName) --environment $(CONTAINERAPPS_ENVIRONMENT) --cpu $(ContainerCpu) --memory $(ContainerMemory) --min-replicas $(MinReplicas) --max-replicas $(MaxReplicas)  --ingress $(IngressType) --target-port $(IngressPort) --registry-server $(RegistryServer) --registry-username $(RegistryUsername) --registry-password $(RegistryPasswordSecretName)  --secrets $(PasswordSecretName)=$(RegistryPassword) --image  $(ContainerImage) --env-vars ENV_ENVIRONMENT_NAME_PREFIX=$(ENV_ENVIRONMENT_NAME_PREFIX) ENV_CLIENT_CONFIG_PATH=$(ENV_CLIENT_CONFIG_PATH)

To summarize it, get all VARIABLES with prefixed name ENV_ and pass it to --env-vars

Important addition: Not all variables are simple strings, some are json blobs and may contain spaces as well as unescaped quotes:

\"{\"type\":\"service_account\",\"project_id\":\"projectID\",\"private_key_id\":\"PRIVATEREGISTRYID\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\nPRIVATEKEYHASH\n-----END PRIVATE KEY-----\n\",\"client_email\":\"publisher@email.iam.gserviceaccount.com\",\"client_id\":\"11132670678102951289\",\"auth_uri\":\"https:\/\/accounts.google.com\/o\/oauth2\/auth\",\"token_uri\":\"https:\/\/oauth2.googleapis.com\/token\",\"auth_provider_x509_cert_url\":\"https:\/\/www.googleapis.com\/oauth2\/v1\/certs\",\"client_x509_cert_url\":\"https:\/\/www.googleapis.com\/robot\/v1\/metadata\/x509\/az-fellipelli-publisher%40codescovery-default.iam.gserviceaccount.com\"}\"

UPDATE

I've done some tests and so far I've ended up with the following approach

formatted_env_vars=$(env | grep '^ENV_' | while read -r line; do
  KEY=$(echo "$line" | cut -d '=' -f 1)
  VALUE=$(echo "$line" | cut -d '=' -f 2-)
  echo "${KEY}=\"${VALUE}\""
done | tr '\n' ' ')
az containerapp create --resource-group $(ResourceGroupName) --name $(ContainerAppName) --container-name $(ContainerAppName) --environment $(CONTAINERAPPS_ENVIRONMENT) --cpu $(ContainerCpu) --memory $(ContainerMemory) --min-replicas $(MinReplicas) --max-replicas $(MaxReplicas)  --ingress $(IngressType) --target-port $(IngressPort) --registry-server $(RegistryServer) --registry-username $(RegistryUsername) --registry-password $(RegistryPasswordSecretName)  --secrets $(PasswordSecretName)=$(RegistryPassword) --image  $(ContainerImage) --env-vars ENVIRONMENT_NAME_PREFIX=$(ENVIRONMENT_NAME_PREFIX) CLIENT_CONFIG_PATH=$(CLIENT_CONFIG_PATH) "$formatted_env_vars"

This command results something like:

az containerapp create --resource-group my-resource-group --name my-app-name --container-name my-app-name --environment my-environment --cpu 0.25 --memory 0.5Gi --min-replicas 0 --max-replicas 1  --ingress external --target-port 80 --registry-server gcr.io --registry-username _json_key --registry-password secretref:registrypassword  --secrets registrypassword={"type":"service_account","project_id":"my-project-id","private_key_id":"ecd532a427a221bb5e95e6433a27723b25413392","private_key":"-----BEGIN PRIVATE KEY-----nMIIEvgIDCBENBhqkgkiG9w0BAQFEABSCBKgwggSkAgEAAoIBAQC0FPeHr6LlOibbnNUUsNzJfKNED7eZ6PT71UesZ8blWn2C2FWP7wI7OrfE19FtF8Jt6RP8J2x2jSafmnuR0FmNH1tkWuq+BenZjaefW/a9LAYzdKspq9Ob4caNYXbBNvJ6xrn+x6+m3TGg6nJXDlAmV+FRYAePdHJ+uanfZUx1e2HfGSswCczMSevTXSNU+6VKnmtL2hg1uam3ywnp2VoMleY2dkrAV2xqAJDnAV21oC57MAi5kEryRYNtzaP40m87Csl0g31VNjn9Acfn4cMsHhoM/LBL5AQwklQstJMVk4dJGfLBOi46G8BES0zuGVrLBY72dUjz3EHiIc3Kn54Fn1eedAgMBAAECggEACiDbwy/oD6kXnri8ST5ZgOykxNnOirWDIv0WggEZCrwwnDpWLmSrSL06gI1wYFVCuyMu/ZHQ7L3fE4lkt3DGeJ943m4WodVnT6sd8GKVwBIHzn0ESl2dooW3TYwRGT5uti6Y9s8y7XmE2d1xdgACFgX/UDpccn8UKXh7y2rBICJuXzn1wG4zIrb9bukysxbC1f55MdqWAHhmGY2EiCRMQ1iABh8cueZffRb25MqN/quqdNPnEJ3eZ5AXo1kgb+JE85f6T8UxGBbV8NhYn4EBHoO3QDS8EPk+QnyBcTN6TWqB+Za3nfTBW878dbljFEtg8cvUJel1xR2QiiVoOYORntlCJaQKBgQD3/Vs0kHfxmElCHpEmn6ResBMDNgQNvtP86825QYjnVobqpujsoT3Oq7HZrFdnrtncE5BwDonLfMzIpNwWnnb3Z0AoZKNL7UK1tWk7uoLLMqcgK6cJ1uPhzZ9jDQLl/kQzF1UMcZ7qKzKMDORGX7nxrW2drV+Hl6sR2vJ+5eNvs7SbwKBgQC55hOb1IQES4QHQakIMh9RumnKG3vILdqXnjux0Vn7BM+2PDHtzXaGIYMyUdyYR2lijf3RTqWOaw6+16AbUmGdGpKZormT7uujfnyQPAvlpIpfNpuQOv8OC3Wd8ORBJ7JqQ2mtO/JAT8D02x8fqJ/oN83SrpeUyOar96npU8TjuV8swKBgQDbJ+a7PFUTh+5t5U+HoITP+19b/YDI7p6EPW8m0LelqQiqSC9Rnf22MBu7b0fLFEXp7Mw/i0mRMBhV1QO/ZOXLoMPb2i0h9fld42pQDmoU3pIvuseken7BsWG+5xfBo4Qn1p3None7IDHKevYdieuXgK6TuZWkN2YqPKW3JizNZANwKBgAvCnpGbtUNS8ukODQeBE23peIqcoifImBHgmEItjlW0/jb3I+l5YBAoOpaKWzw++EnF1ndLR+ONDnpEwLu7LjWVU8wVJKBhVuw6C0pMoiwK9r4DRGAhe1U2IGC3hpkgm8bhMOnRGsJKk+P4EhrmH/2W/cgPsC6bb9svWRS5OfF2HrFAoGBANG612EferWkG7jLrVpzniYabgP73yjRAjt+J3JLV3QW42JyXEh+fgJwU4A6mty9GxcNPogTsdz16PdFOG1YtndKHCZUP29i1hIG4BxgxxXjWDrfzH+mEdNg+80v2R5YTM0L2vHUMMU63VR603fl5En4OfBG5cnJ9A3RDeg6mRIGEMnn-----END PRIVATE KEY-----n","client_email":"publisher@registry.iam.gserviceaccount.com","client_id":"2222035690768105943154","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/publisher%registry.iam.gserviceaccount.com"} --image  gcr.io/registry/codescovery/my-site-vue3:0.1.245274 --env-vars ENVIRONMENT_NAME_PREFIX=ENV_ CLIENT_CONFIG_PATH=/app/client/configurations/settings.json ENV_SITE_CREATOR_NAME="Example Creator" ENV_SITE_NAME="Site Name" ENV_SITE_TITLE="My Site Title"

But apparently it's writing the dynamically variables into ENV_SITE_CREATOR_NAME only:

Here are the environment variables in azure

What I'm missing?

Update 2

According to @siddheshdesai Answer I got the following result

Answer 1

You can add the Environment variables in the Variables tab of Azure Release pipeline and then use it as an ENV_VAR in your pipeline, Refer below:-

And I used the same code as an environment variable in the inline script :-

az config set extension.use_dynamic_install=yes_without_prompt
az containerapp create --name $(ENV_ContainerAppName)  --resource-group $(ENV_ResourceGroupName) --cpu $(ENV_ContainerCpu) --memory $(ENV_ContainerMemory) --environment $(ENV_CONTAINERAPPS_ENVIRONMENT)

And the container app got deployed successfully like below:-

You can also add the same variables in the Variable group and use those variable group across multiple pipeline within your Azure DevOps project, Refer below:-

And reference it in any pipeline like below:-

In order to call the Environment variables dynamically, you can make use of below CLI commands in your Azure CLI task:-

Code 1:-

#!/bin/bash

# set the prefix for your environment variables
envPrefix="ENV_"

# get the list of environment variables that have the specified prefix
envVars=$(env | grep "^$envPrefix" | sed "s/^$envPrefix//")

# create a string of environment variables in the format "KEY=VALUE"
envVarsStr=""
for envVar in $envVars; do
  envVarVal=$(eval "echo \$$envVar")
  envVarsStr+="$envVar=$envVarVal "
done

# create a string with the environment variables argument for the az command
envVarsArg="--env-vars $envVarsStr"

# set the extension configuration
az config set extension.use_dynamic_install=yes_without_prompt

# create the container app with the specified environment variables
az containerapp create --name $ContainerAppName --resource-group $ResourceGroupName --cpu $ContainerCpu --memory $ContainerMemory --environment $CONTAINERAPPS_ENVIRONMENT $envVarsArg

Code 2:-

#!/bin/bash

# Set the prefix for your environment variables
envPrefix="ENV_"

# Retrieve environment variables dynamically
containerAppName=${!envPrefix"CONTAINERAPP_NAME"}
resourceGroupName=${!envPrefix"RESOURCE_GROUP_NAME"}
containerCpu=${!envPrefix"CONTAINER_CPU"}
containerMemory=${!envPrefix"CONTAINER_MEMORY"}
containerAppsEnvironment=${!envPrefix"CONTAINERAPPS_ENVIRONMENT"}

# Retrieve environment variables that start with the prefix
envVars=()
while IFS='=' read -r key value; do
  envVars+=("$key=$value")
done < <(env | grep "^$envPrefix")

# Join the environment variables with a space
envVarsArg="--env-vars ${envVars[*]}"

# Set the Azure CLI extension configuration
az config set extension.use_dynamic_install=yes_without_prompt

# Create the Azure Container App
az containerapp create \
  --name "$containerAppName" \
  --resource-group "$resourceGroupName" \
  --cpu "$containerCpu" \
  --memory "$containerMemory" \
  --environment "$containerAppsEnvironment" \
  $envVarsArg

Above codes gets the required environment variables dynamically and sets them to the desired variables. Then it retrieves the variable that start with "ENV_" and joins them with space and creates envVarsArg variable. Then it uses Azure CLI extension config to create Azure container app with retrieved variable.

Output:-

Updated code:-

# set the prefix for your environment variables
envPrefix="ENV_"

# get the list of environment variables that have the specified prefix
envVars=$(env | grep "^$envPrefix" | sed "s/^$envPrefix//")

# create a string of environment variables in the format "KEY=VALUE"
envVarsStr=""
for envVar in $envVars; do
  envVarVal=$(eval "echo \$$envVar")
  
  # if the value of the environment variable contains spaces, enclose it in quotes
  if [[ "$envVarVal" == *" "* ]]; then
    envVarVal="\"$envVarVal\""
  fi
  
  envVarsStr+="$envVar=$envVarVal "
done

# create a string with the environment variables argument for the az command
envVarsArg="--env-vars $envVarsStr"

# set the extension configuration
az config set extension.use_dynamic_install=yes_without_prompt

# create the container app with the specified environment variables
az containerapp create --name $ContainerAppName --resource-group $ResourceGroupName --cpu $ContainerCpu --memory $ContainerMemory --environment $CONTAINERAPPS_ENVIRONMENT $envVarsArg

Updated code 2:-

# set the prefix for your environment variables
envPrefix="ENV_"

# get the list of environment variables that have the specified prefix
envVars=$(env | grep "^$envPrefix" | sed "s/^$envPrefix//")

# create an array of environment variables and their values
declare -a envVarsArr
for envVar in $envVars; do
  envVarVal=$(eval "echo \$$envVar")
  
  # if the value of the environment variable contains spaces, enclose it in quotes
  if [[ "$envVarVal" == *" "* ]]; then
    envVarVal="\"$envVarVal\""
  fi
  
  envVarsArr+=("$envVar=$envVarVal")
done

# join the elements of the array with spaces to create the envVarsStr variable
envVarsStr="${envVarsArr[@]}"

# create a string with the environment variables argument for the az command
envVarsArg="--env-vars $envVarsStr"

# set the extension configuration
az config set extension.use_dynamic_install=yes_without_prompt

# create the container app with the specified environment variables
az containerapp create --name $ContainerAppName --resource-group $ResourceGroupName --cpu $ContainerCpu --memory $ContainerMemory --environment $CONTAINERAPPS_ENVIRONMENT $envVarsArg

Answer 2

Try this

#!/bin/bash

# Get all environment variables with prefix "ENV_"
# Initialize variable
ENV_VARS=""

# Loop through all environment variables
while IFS= read -r var
do
  # Check if the variable starts with ENV_
  if [[ "$var" == ENV_* ]]; then
    # Extract the variable name and value
    name=$(echo "$var" | cut -d '=' -f 1 | sed 's/^//')
    value=$(echo "$var" | cut -d '=' -f 2-)

    # Escape any single quotes in the value
    value="${value//\'/\'\\\'\'}"

    # Append the variable and its value to the result
    ENV_VARS+=" $name='$value'"
  fi
done <<< "$(env)"

# Create the container app in Azure
echo "Creating container app $CONTAINERAPP_NAME in Azure..."
az containerapp create \
  --name $CONTAINERAPP_NAME \
  --resource-group my_resource_group \
  --image $IMAGE_NAME \
  --env-vars $ENV_VARS

Answer 3

In powershell, you can grab all the environment variables and then buid the string with which you call the azure cli:

gci env:*

Filtering that down to all variables starting with ENV_:

gci env:* | ?{ $_.name -like "ENV_*" }

That leaves us with building the string to pass to azure-cli:

((gci env:* | ?{ $_.name -like "ENV_*" }) | %{ "$($_.name)=$($_.value)" }) -join " "

And then we're left with the quoting issue. I'm not 100% sure how azure-cli wants its quotes, "abc=def ghi" or abc="def ghi", but that's easy to add in:

# either: "abc=def ghi"
((gci env:* | ?{ $_.name -like "ENV_*" }) | %{ "`"$($_.name)=$($_.value)`"" }) 

# Or abc="def ghi"
((gci env:* | ?{ $_.name -like "ENV_*" }) | %{ "$($_.name)=`"$($_.value)`"" })

Then call azure-cli:

& az containerapp create `
   --resource-group $(ResourceGroupName) `
   --name $(ContainerAppName) `
   --container-name $(ContainerAppName) `
   --environment $(CONTAINERAPPS_ENVIRONMENT) `
   --cpu $(ContainerCpu) --memory $(ContainerMemory) `
   --min-replicas $(MinReplicas) --max-replicas $(MaxReplicas)  `
   --ingress $(IngressType) `
   --target-port $(IngressPort) `
   --registry-server $(RegistryServer) --registry-username $(RegistryUsername) --registry-password $(RegistryPasswordSecretName) `
   --secrets "$(PasswordSecretName)=$(RegistryPassword)" `
   --image  $(ContainerImage) `
   --env-vars ((gci env:* | ?{ $_.name -like "ENV_*" }) | %{ "`"$($_.name)=$($_.value)`"" })

Or build the whole string and then calling it through invoke-expression:

$envVars = ((gci env:* | ?{ $_.name -like "ENV_*" }) | %{ "`"$($_.name)=$($_.value)`"" }) -join " "
$command = "& az containerapp create `
   --resource-group $(ResourceGroupName) `
   --name $(ContainerAppName) `
   --container-name $(ContainerAppName) `
   --environment $(CONTAINERAPPS_ENVIRONMENT) `
   --cpu $(ContainerCpu) --memory $(ContainerMemory) `
   --min-replicas $(MinReplicas) --max-replicas $(MaxReplicas)  `
   --ingress $(IngressType) `
   --target-port $(IngressPort) `
   --registry-server $(RegistryServer) --registry-username $(RegistryUsername) --registry-password $(RegistryPasswordSecretName) `
   --secrets `"$(PasswordSecretName)=$(RegistryPassword)`" `
   --image $(ContainerImage) `
   --env-vars $envVars"

iex $command # invoke expression

To pass variables to Powershell in a safer manner, you can use:

- powershell: |
      & az .... $env:RegistryUsername
    env:
      RegistryUsername: $(RegistryUsername)

Or in UI based pipelines and releases:

This usually simplifies many of the quoting issues since PowerShell will know how to pass the contents of the variable safely to the invocation of az, where as when you inline the azure pipelines variable reference, the contents will be added to the script without context. I'm not aware of any clear docs on the dangers, but here's a doc from the GitHub Actions team that explains the potential issue. This applies to Azure Pipelines as well.