Unit tests sometimes failing, sometimes passing

Question

I have a mocha unit test suite.

When I run them locally, everything works fine. When I run them on our Jenkins CI server, they sometimes fail, sometimes pass.

I'm just not able to reproduce why they fail. What might cause this behavior?

expected files being temporarily locked? running into issuees with multi-threading tests on a high-powered CI that doesn't occur on dev machines? temporary network glitches blocking things? there are many things that can cause hard-to-track unit test failures... it's all part of problem solving. is it the same tests that prove to be problematic? — TZHX, Sep 27 '12 at 14:47

score 12 · Answer 1 · answered Sep 29 '12 at 01:21

Tests can fail intermittently for a number of reasons and identifying why they fail is often revealing about your codebase and environment.

Here are a few possible causes:

• Shared objects - singletons that hold state can cause problems between tests if the test environment isn't reset to a well known state. if if your test runner executes tests in a non-deterministic order you may see random errors that are actually exposing corrupted state issues

• Environmental and external dependencies - any external object that can hold state can cause unpredictable results

• Timing - sometimes tests are written with timeouts or thread sleeps that are too specific. If the build server is operating under heavy load these timeouts may not be long enough

As general guidance, tests must be:

isolated: tests focus on one unit at a time
repeatable: produces the same results each time
independent: the order in which tests are executed should not matter

score 4 · Answer 2 · answered Sep 30 '12 at 13:50

I would try to narrow down the problem by reducing the number of executors to 1. If the tests are still failing intermittently, then you have a Test Run War, otherwise (given they run fine locally) it looks like a Resource Leakage.

More info at http://xunitpatterns.com/Erratic%20Test.html

score 2 · Answer 3 · answered Sep 27 '12 at 15:11

I would geuss you are reusing the test fixtures for multiple tests. In some cases the order of the tests is changed from the order on your local machine and you do not have a correct fixture when you start a test. To fix it, you should give each test a clean fixture.

An other cause can be that you reuse a database from multiple locations when runnen tests (two build servers with the same test configuration). Give each build server its own database. Or sub/mock your database layer. Your test should not depend on external resources except when you are testing the connections to your external resources.

Sadly, No. The whole database is wiped each test is run. I'm using mocha to run the tests, so they are run in series. The database is installed locally on that machine, so that should not make other services use it aswell. — Sebastian Hoitz, Sep 27 '12 at 15:16

Justinas Jakavonis · Answer 4 · 2017-06-14T10:33:53.163

1

It may happen if you're testing multithreaded code and use Thread.sleep(time); in your tests. If sleep time is not long enough, tests may pass or fail depending on computers processing speed. Related discussions:

edited Jun 14 '17 at 10:33

answered May 25 '17 at 08:01

Justinas Jakavonis

8,220
10
69
114

score 0 · Answer 5 · answered Mar 04 '22 at 19:30

I'm working with jest. My problem was the mock data causing the tests to fail intermittently. Clearing mocks and using fake timers inside each describe helped my problem:

describe("my test", ()=> {

    beforeAll(() => {
        jest.setTimeout(30000);
        jest.useFakeTimers();
      });
    afterAll(() => {
        jest.useRealTimers();
      });
    beforeEach(() => {
        jest.clearAllMocks();
      });

    it("my test", async()=>{....})


    ...

})

Unit tests sometimes failing, sometimes passing

5 Answers5