We've TestCafe.js UI tests that runs regression suite on Jenkins environment.
We're exploring a way to create a mechanism, wherein we can potentially set certain pass threshold for the test suite to make the Jenkins job status as Pass / Fail.
i.e. if 98% + tests pass then mark the test job as pass.
Under XUnit projects same could be achieved using XUnit test Plugin etc. Example reference: How can I have Jenkins fail a build only when the number of test failures changes?
How to fail a Jenkins job based on pass rate threshold of testng tests
How to not mark Jenkins job as FAILURE when pytest tests fail
Is similar possible for TestCafe based tests either through TestCafe customization / through some Jenkins plugin?
Our Jenkins file:
#!groovy
pipeline {
environment {
CI = 'true'
}
options {
buildDiscarder(logRotator(numToKeepStr: '50'))
disableResume()
ansiColor('xterm')
}
agent none
// Define the stages of the pipeline:
stages {
stage('setup') {
steps {
script {
cicd.setupBuild()
}
}
}
// Use the make target to run tests:
stage('Tests') {
agent any
steps {
script {
cicd.withSecret(<keys>) {
cicd.runMake("test")
}
}
}
post {
cleanup {
archiveArtifacts artifacts: "screenshots/**", allowEmptyArchive: true
}
}
}
}
post {
success {
script { cicd.buildSuccess() }
}
failure {
script {
slackSend channel: "#<test-notifications-channel>", color: 'bad', message: "Regression tests failed or unstable <${env.RUN_DISPLAY_URL}|${env.JOB_NAME}>"
cicd.buildFailure()
}
}
}
}
enter code here
