Enumerate secret variables in Azure Pipelines

Question

I have a build step in Azure Pipelines that takes the variables from Azure Pipelines and uploads them somewhere equally secret. Currently I have about 50 builds, and each build has anywhere between 5-20 variables.

Some are secret and some are not. So for non secret ones I enumerate all the set ones and off i go; but for secret ones I need to add them to the build step manually; further, because I am writing them with the same keys i need to:

Declare variable in the group e.g. MyPrefix.MyVar
Edit the build step to say /specialtool --vars=MyPrefix.MyVar=$(MyPrefix.MyVar) which is rather mundane.

I found that I can get a list of variables using the Azure DevOps api, so i thought i could just modify the next build step as the build is running.

However, if I update the same build definition that is currently running (to dynamically write the command), it is not sent to the agent (rather, it feels like all arguments for tasks are captured when the whole build is triggered). Any thoughts on how i can dynamically enumerate secret vars to feed to my tool?

Do you mean you want to update the variables during the build running? — Eddie Chen - MSFT, Sep 05 '18 at 07:24
@EddieChen-MSFT Correct. Ideally, i want some method that allows me to read secrets into my code without knowing their names BEFORE the build starts (e.g. if someone attaches another variable group to the same build, i want to be able to read secrets from that); in absence of that, if i can at least have one step read all variable names and then update the OTHER step invocation, that'd solve it for me. — zaitsman, Sep 05 '18 at 07:57
Programmatically altering the pipeline definition sounds overengineered and brittle. You mention the Azure DevOps API. Write a PowerShell script that a.) invokes the API to get the list of variables (as you've already learned); b.) loops through them and builds up the command line you describe in step 2; c.) invokes the special tool that archives your secrets. That is, do it all in one PS script/build task. If c.) can't immediately follow b.), then store result of b.) in an intermediate place (enviroment var, git repo, encrypted disk file, etc.) and read it back as part of c.) — Richard II, Jun 26 '19 at 14:16

score 0 · Answer 1 · answered Sep 05 '18 at 08:09

0

You can use VSTS Logging Commands to update variable value during the build. This will make the updated variable to be available in next build task.

Write-Host "##vso[task.setvariable variable=testvar;]testvalue"

answered Sep 05 '18 at 08:09

Eddie Chen - MSFT

29,708
2
46
60

But that won't let me read the secret values anyway? E.g. even if i did output the names of variables in the variable group, this won't let me read the actual values. – zaitsman Sep 05 '18 at 08:12
@zaitsman Yes, secret variable cannot be read via script since they are not passed into tasks as variables. It seems that I misunderstood your question. Do you want to update the variable value during the build or update the variable argument for the build task? – Eddie Chen - MSFT Sep 05 '18 at 08:18
i want to update arguments passed to my task. E.g. if if i pass secret variables to a task by name, i want to add those names in the previous step. This way i was hoping to access all secret variables in the next step. – zaitsman Sep 05 '18 at 08:30
@zaitsman There isn't any way to achieve this for now. But as a workaround, you can create two build definition to do this. The first build definition get all the variables and update the second build definition to pass more secret variables. The second one run the things you'd like to do. And set the first build definition to trigger the second one. – Eddie Chen - MSFT Sep 05 '18 at 08:42
i have about 60 build definitions doing this; doubling them would bring it to unwieldy 120 – zaitsman Sep 05 '18 at 10:48

Gopal Krishnan · Answer 2 · 2019-04-11T16:16:05.917

So I have been looking at a solution for this too. It appears the only way to do this at the moment is to write a custom task. Within a custom task you can get hold of secret values dynamically.

An example is the 'vsts-replacetokens-task' (https://github.com/qetza/vsts-replacetokens-task/blob/master/task/index.ts)

Internally it uses the vsts task library (vsts-task-lib/task) (https://github.com/Microsoft/azure-pipelines-task-lib/blob/master/node/task.ts)

This vsts task library exposes methods like GetVariables() and GetVariable() etc. which can provide what you need. Unfortunately bit long winded, but the only way that I can see.

jessehouwing · Answer 3 · 2019-04-10T11:43:52.420

When you create a Typescript custom task (NodeJS based), you can access all the build variables that are available to the build at that point in time through the getVariable api.

This function returns an array of VariableInfo:

/** Snapshot of a variable at the time when getVariables was called. */
export interface VariableInfo {
    name: string;
    value: string;
    secret: boolean;
}

When you create a PowerShell3 custom task, you can access all the build variables that are available to the build at that point in time through the Get-VstsTaskVariable function.

Which returns a similar object structure as the Node version:

New-Object -TypeName psobject -Property @{
    Name = $info.Name
    Value = Get-TaskVariable -Name $info.Name
    Secret = $info.Secret
}

If you need to support TFS 2015 and the 1.x build agents as well, you can use (now deprecated) PowerShell handler and enumerate the secrets using a custom powershell function I describe here.

Each task SDK (Typescript and Powershell), supports a function to set variables as well. Here is an example of setting the variable value in Typescript:

tl.setVariable(variable, value, isSecret);

And on PowerShell3:

Set-VstsTaskVariable -name $VariableName -value $Value -Secret $IsSecret

And on PowerShell (deprecated):

Write-Host "##vso[task.setvariable variable=$($VariableName);issecret=$($IsSecret)]$Value"

My suspicion is that you'd want to create a single task that reads the variables and invokes the command you mentioned in your original post to then post these variables to the other secret store. It's not recommended to read all the secrets and either store them in non-secret variables or to somehow pass them along to the next task.

It does. The `VariableInfo.secret` will be true in case of a secret. These functions only work in the context of a custom task extension, since they require the additional privileges. And there is a minimum build agent version of 2.205 or something for the Node and PowerShell3 handler, but that is only important if you're targeting very old versions of TFS (read 2015u2 or something). — jessehouwing, Apr 10 '19 at 11:27

Enumerate secret variables in Azure Pipelines

3 Answers3