Get exit code from `az vm run-command` in Azure pipeline

Question

I'm running a rather hefty build in my Azure pipeline, which involves processing a large amount of data, and hence requires too much memory for my buildagent to handle. My approach is therefore to start up an linux VM, run the build there, and push up the resulting docker image to my container registry.

To achieve this, I'm using the Azure CLI task to issue commands to the VM (e.g. az vm start, az vm run-command ... etc).

The problem I am facing is that az vm run-command "succeeds" even if the script that you run on the VM returns a nonzero status code. For example, this "bad" vm script:

az vm run-command invoke -g <group> -n <vmName> --command-id RunShellScript --scripts "cd /nonexistent/path"

returns the following response:

{
  "value": [
    {
      "code": "ProvisioningState/succeeded",
      "displayStatus": "Provisioning succeeded",
      "level": "Info",
      "message": "Enable succeeded: \n[stdout]\n\n[stderr]\n/var/lib/waagent/run-command/download/87/script.sh: 1: cd: can't cd to /nonexistent/path\n",
      "time": null
    }
  ]
}

So, the command succeeds, presumably because it succeeded in executing the script on the VM. The fact that the script actually failed on the VM is buried in the response "message"

I would like my Azure pipeline task to fail if the script on the VM returns a nonzero status code. How would I achieve that?

One idea would be to parse the response (somehow) and search the text under stderr - but that sounds like a real hassle, and I'm not sure even how to "access" the response within the task.

Bright Ran-MSFT · Answer 1 · 2021-03-26T02:29:38.050

2

Have you enabled the option "Fail on Standard Error" on the Azure CLI task? If not, you can try to enable it and run the pipeline again to see if the error "cd: can't cd to /nonexistent/path" can make the task run failed.

If the task still is passed, the error "cd: can't cd to /nonexistent/path" should not be a Standard Error. In this situation, you may need to add more command lines in your script to monitor the output logs of the az command. Once there is any output message shows error, execute "exit 1" to exit the script and return a Standard Error to make the task be failed.

edited Mar 26 '21 at 02:29

answered Mar 26 '21 at 02:19

Bright Ran-MSFT

5,190
1
5
12

Thanks for the suggestion, I can't believe I missed that field. I have set it to true, but the task still succeeds with a `stderr` in the response message: `"message": "Enable succeeded: \n[stdout]\n\n[stderr]\n/var/lib/waagent/run-command/download/116/script.sh: 1: cd: can't cd to /does/not/exist\n",` – mchristos Mar 29 '21 at 09:33
I think its because there is nothing in the `stderr` stream for the command itself, i.e. `az vm run-command` - rather it is buried it the message property of the json response – mchristos Mar 29 '21 at 09:35
Hi @ChrisMarais, Yes, I agree your guess. In short, it is not a Standard Error. So, as I mentioned previously, you need to check the respond message of the `az vm run-command` to see if it contains any failures or errors. – Bright Ran-MSFT Mar 30 '21 at 06:30

score 0 · Answer 2 · answered Oct 13 '21 at 10:25

I solved this by using the SSH pipeline task - this allowed me to connect to the VM via SSH, and run the given script on the machine "directly" via SSH.

This means from the context of the task, you get the status code from the script itself running on the VM. You also see any console output inside the task logs, which was obscured when using az vm run-command.

Here's an example:

  - task: SSH@0
    displayName: My VM script
    timeoutInMinutes: 10
    inputs:
      sshEndpoint: <sshConnectionName> 
      runOptions: inline
      inline: |
        echo "Write your script here"

Not that the SSH connection needs to be set up as a service connection using the Azure pipelines UI. You reference the name of the service connection you set up in yaml.

Get exit code from `az vm run-command` in Azure pipeline

2 Answers2