Why does Pester in Classic Azure Pipeline always fail?

Question

I have a classic build pipeline (not yaml, yet!) with three tasks

Install Pester -> works as expected
Run Pester tests -> always fails!
Publish test results -> works as expected

My script for "Run Pester tests"

Import-Module Pester -Force

$config = [PesterConfiguration]::Default
$config.TestResult.Enabled = $true
$config.TestResult.OutputFormat = "NUnitXml"
$config.TestResult.OutputPath = "$env:SYSTEM_DEFAULTWORKINGDIRECTORY\TEST-results.xml"
$config.Run.Path = ".\my\path*"
$config.Run.PassThru = $true

$results = Invoke-Pester -Configuration $config
if($results.Result -eq "Passed")
{
    Write-host "All tests passed"
    Write-Host "##vso[task.complete result=Succeeded;]DONE"
    exit 0
}

Sample output in Azure DevOps is

Tests completed in 11.43s
Tests Passed: 67, Failed: 0, Skipped: 0 NotRun: 0
All tests passed
Finishing: Run Pester Unit Tests

However, the task always fails

I know from robocopy that Powershell Tasks in Azure will check $LASTEXITCODE and that you are able to overwrite that via Write-Host "##vso[task.complete result=Succeeded;]DONE" and exit 0. Why is this not working here?

What else have I tried?

Activate SilentlyContinue -> results in Warning
Activate Continue on Error -> results in Warning
Activate Ignore $LASTEXITCODE -> results in Failure
Activate Fail on Standard Error -> results in Failure

Did you try to put only 1 test or even without tests and check if it failed? maybe it's a specific test that cause it. — Shayki Abramczyk, Jun 06 '23 at 07:07
That's a good point. I can't remember if I tried executing a single test. I will double check that and come back to you. — tomwaitforitmy, Jun 06 '23 at 09:51
@ShaykiAbramczyk thank you! You helped me out a lot here! Can I somehow give you the bounty? — tomwaitforitmy, Jun 06 '23 at 12:56
I glad it's helped you! your learning below # 1 it's exactly what I thought! Hmm I guess I can't get the bounty because I didn't post it as an answer, but nvm thanks! — Shayki Abramczyk, Jun 06 '23 at 13:00

score 0 · Accepted Answer · answered Jun 06 '23 at 12:55

Oh boy this was a tough one: The comment by Shayki Abramczyk enabled me to track it down. Is it possible to grant him the bounty somehow?

The cause was a unit test, that was changing the Azure DevOps task output via Write-Host "##vso[task.complete result=Failed;]DONE". Since my Powershell scripts are supposed to work in Azure DevOps, they should do things like this. The test was named nicely for Pester it "Should print out error if build requirement not found"

My learning here

Make your issues smaller. Reducing this to a single test, showed me directly that one of my tests was the cause even though they all passed.
You cannot test the following line in Azure DevOps with Pester, because if will always make your tests fail Write-Host "##vso[task.complete result=Failed;]DONE"
A test executing that line, will show "Passed" which can be correct, but is annoying and hard to find.
You cannot overwrite that by the following line afterwards Write-Host "##vso[task.complete result=Succeeded;]DONE"

score 0 · Answer 2 · answered Jun 07 '23 at 14:07

The relevant documentation is "Azure DevOps / Azure Pipelines / Logging commands / Complete: Finish timeline"

##vso[task.complete]current operation

result =

Succeeded: The task succeeded.

SucceededWithIssues: The task ran into problems. The build will be completed as partially succeeded at best.

Failed: The build will be completed as failed. (If the Control Options: Continue on error option is selected, the build will be completed as partially succeeded at best.)

That means when you use the ##vso[task.complete] logging command in Azure DevOps, it directly controls the outcome of the task. If your unit test executes a line that sets the task result to Failed, then the task will fail, even if all your tests pass.

To prevent this, you might want to consider a different approach to testing that line of code. Instead of actually running the Write-Host "##vso[task.complete result=Failed;]DONE" line in your test, you could mock the Write-Host function to just return a specific value or message when it is called with that argument.
That way, you can test that your script is correctly calling Write-Host with the expected argument, without actually affecting the outcome of the task.

A mock Write-Host in your test could look like:

Describe "Test Write-Host" {
    It "Should print out error if build requirement not found" {
        # Mock Write-Host to just return the message instead of printing it
        Mock Write-Host { return $args[0] }

        # Run your script or function that's supposed to call Write-Host with the failure message
        $result = Run-YourScript-Here

        # Check that Write-Host was called with the expected argument
        Assert-MockCalled Write-Host -ParameterFilter { $args[0] -eq "##vso[task.complete result=Failed;]DONE" }
    }
}

This way, you can verify that your script is correctly calling Write-Host with the failure message, without actually causing the task to fail.

You script becomes:

Import-Module Pester -Force

$config = [PesterConfiguration]::Default
$config.TestResult.Enabled = $true
$config.TestResult.OutputFormat = "NUnitXml"
$config.TestResult.OutputPath = "$env:SYSTEM_DEFAULTWORKINGDIRECTORY\TEST-results.xml"
$config.Run.Path = ".\my\path*"
$config.Run.PassThru = $true

# Mock Write-Host to just return the message instead of printing it
Mock Write-Host { return $args[0] }

$results = Invoke-Pester -Configuration $config

if($results.FailedCount -eq 0)
{
    Write-host "All tests passed"
    Write-Host "##vso[task.complete result=Succeeded;]DONE"
    exit 0
}

This will mock Write-Host for all the tests run by Invoke-Pester.
If you want to mock it for a specific test, you'll need to include the mock inside the Describe block of that test.

Thank you for even more details and a way to mock the output. I would have never guessed that mocking is so easy in Powershell. — tomwaitforitmy, Jun 09 '23 at 06:56

Why does Pester in Classic Azure Pipeline always fail?

2 Answers2

`##vso[task.complete]current operation`