19

How to interrupt all Cypress tests on the first test failure?

We are using semaphore to launch complete e2e tests with Cypress for each PR. But it takes too much time.

I'd like to interrupt all tests on the first test failure.

Getting the complete errors is each developer's business when they develop. I just want to be informed ASAP if there is anything wrong prior to deploy, and don't have to wait for the full tests to complete.

So far the only solution I came up with was interrupting the tests on the current spec file with Cypress.

afterEach(() => {
        if (this.currentTest.state === 'failed') {
            Cypress.runner.end();
        }
    });

But this is not enough since it only interrupts the tests located on the spec file, not ALL the other files. I've done some intensive search on this matter today and it doesn't seem like this is a thing on Cypress.

So I'm trying other solutions.

1: with Semaphore

fail_fast:
  stop:
    when: "true"

It is supposed to interrupt the script on error. But it doesn't work: tests keep running after error. My guess is that Cypress will throw an error only when all tests are complete.

2: maybe with the script launching Cypress, but I'm out of ideas Right now here are my scripts

"cy:run": "npx cypress run",
"cy:run:dev": "CYPRESS_env=dev npx cypress run",
"cy:test": "start-server-and-test start http-get://localhost:4202 cy:run"
Florent Arlandis
  • 866
  • 1
  • 10
  • 29
  • Getting error at runner Cypress.runner.end(); as Property 'runner' does not exist on type 'Cypress & CyEventEmitter' – ksk Nov 01 '22 at 10:40

5 Answers5

16

EDIT: It seems like this feature was introduced, but it requires paid version of Cypress (Business Plan). More about it: Docs, comment in the thread

Original answer:

This has been a long-requested feature in Cypress for some reason still has not been introduced. There are some workarounds proposed by the community, however it is not guaranteed they will work. Check this thread on Cypress' Github for more details, maybe you will find a workaround that works for your case.

P D
  • 752
  • 1
  • 8
  • 13
  • 1
    Thank you. This is exactly what I thought, I visited this link before posting my question. Workarounds are okay(ish) but I was wondering if this could be achieved with Semaphore or npm scripts – Florent Arlandis May 08 '20 at 17:22
  • To my knowledge, it is not possible. All the workarounds I have encountered are in this link. If my answer has helped you, please consider marking it as accepted answer. – P D May 09 '20 at 17:52
  • But Cypress.io though provided this feature its useless some times in case if we want to skip only the spec file its which are failing but not Others spec files – ksk Nov 02 '22 at 07:26
  • Its available on the dashboard with the paid version. [Source](https://github.com/cypress-io/cypress/issues/518#issuecomment-809514077) – wamster Jan 23 '23 at 22:17
8

The solution by @user3504541 is excellent! Thanks a ton. I already started giving up on using Cypress since these issues keep popping up. But in any case, here's my config:

support/index.ts

declare global {
  // eslint-disable-next-line
  namespace Cypress {
    interface Chainable {
      interrupt: () => void
    }
  }
}

function abortEarly() {
  if (this.currentTest.state === 'failed') {
    return cy.task('shouldSkip', true)
  }
  cy.task('shouldSkip').then(value => {
    if (value) return cy.interrupt()
  })
}

commands/index.ts

Cypress.Commands.add('interrupt', () => {
  eval("window.top.document.body.querySelector('header button.stop').click()")
})

In my case the Cypress tests were left pending indefinitely on the CI (Github action workflow) but with this fix they interrupt properly.

TeemuK
  • 2,095
  • 1
  • 18
  • 17
7

A little hack that worked for me

Cypress.Commands.add('interrupt', () => {
  eval("window.top.document.body.querySelector('header button.stop').click()");
});
user3504541
  • 81
  • 1
  • 2
  • 1
    While it not a great solution because like you said, it is a bit hacky, it works great. Maybe it would be better to send the keyboard shortcut for the stop button (the s key) which is less likely to change in future versions of Cypress. Also, I might be wrong but I think you don't need the `eval()` here. – Gfra54 Mar 04 '21 at 16:18
1

Options with Cypress.runner.end() or Cypress.runner.stop() don't work for me as they stop cypress before uploading test results. After some puzzling I came up with the following. This requires no extra setup, magic variables etc, just a simple afterEach call.

If the current test is failed, it cancels all subsequent tests within the same container. Since I only use one level of nesting (ie describe( it(), it(), it(), ...)), this is enough. All tests (within the spec) after a failed are now skipped.

This uses the Mocha bail option.

Enables or disables bailing on the first failure.

Works either on the global Cypress e2e.ts (or .js) level, or in individual specs:

before(() => {
  // Skip subsequent tests in spec when one fails.
  (cy.state('runnable').ctx as Mocha.Context).currentTest.parent.bail(true);
});
w5l
  • 5,341
  • 1
  • 25
  • 43
0

This is available as the Auto Cancelation feature, which is part of Smart Orchestration, but is only available to Business Plan. From the Auto Cancelation docs:

Continuous Integration (CI) pipelines are typically costly processes that can demand significant compute time. When a test failure occurs in CI, it often does not make sense to continue running the remainder of a test suite since the process has to start again upon merging of subsequent fixes and other code changes. When Auto Cancellation is enabled, once the number of failed tests goes over a preset threshold, the entire test run is canceled. Note that any in-progress specs will continue to run to completion.

wamster
  • 171
  • 12