0

I am not sure what is the best practice for running unit tests.

I suppose every developers should pass the unit tests locally before pushing the code to the GIT repo. And then the CI server (Jenkins) would pick up the new changes and run the tests again. Why do we want to do it twice? Should we?

If the unit tests take a lot of time to run, do we expect the developer only picks the tests related to the change or runs every tests (even outside the scope of his projects), assuming we have a big maven multi-module POM.

Also consider we usually have a powerful hardware for CI server and relatively less powerful workstation for developers.

Frankie Hung
  • 561
  • 1
  • 6
  • 16
  • 1
    First what do you mean by `we have big multi module build`? How long does it take to build on your CI server? How long take the tests only to run? Do you have unit and integration tests? Have you correctly separated them? Do you use parallel build on your ci system? Do you use branches to develop? – khmarbaise Jul 06 '18 at 06:01

2 Answers2

4

If the unit tests take a lot of time to run, do we expect the developer only picks the tests related to the change or runs every tests (even outside the scope of his projects), assuming we have a big maven multi-module POM.

As a developer changes a class, modifies the database structure or makes any change that could have side effects, he/she will not/cannot know all potential side effects on the whole application.
And he/she never has to try to be too clever by saying : "I know that it may have be broken with my changes, so I will run just this test".

Unit testing ensures a some degree of code quality : don't make it less helpful

The unit tests are also non-regression tests. To not play all non regression tests before commit and push is taking the risk to introduce flawed code in the source content management.
You will never do it.

Unit tests have to be fast executed

If units test are so long to be executed as it hurts the developer velocity , it generally means that they are bad designed or maybe that they are not real unit tests. A unit test is designed to be run fast.
If you write tests that are long to be run because they require to start a server, to load/clean data in a database, to load/unload some containers, and so for... it means you didn't write unit test but integration tests. These tests are not designed to be executed regularly and automatically on the local development machine but on a CI tool.

The CI has to run all tests

Do we need to run tests in CI server if every developers run the tests before push?

As explained, integration tests have to be executed by the CI tool.
About unit testing, sparing their execution in the CI side is not a good idea either.
Of course developers have to run the tests before pushing to the SCM but actually you don't have the guarantee that it will always be done.
Besides, even in a perfect world where developers execute all tests before pushing, you could fall into cases where the tests are successful on the developer machine but fail on the CI or the other developer machines.
For example, the developer could introduce in the base code some absolute paths specific to its machine or he/she could also forget to replicate a modification on the database used in the CI environment.
So running all tests (unit and integration tests) has not to be an option for the CI.

davidxxx
  • 125,838
  • 23
  • 214
  • 215
  • I don't understand the statement "unit tests are non-regression tests". They may serve many purposes but regression should be one of those. – Frankie Hung Jul 06 '18 at 06:11
  • And it is ! I think that it is a semantic issue :) Non-regression testing doesn't mean "not for regression testing" but "to get no regression" . https://en.wikipedia.org/wiki/Non-regression_testing – davidxxx Jul 06 '18 at 06:22
  • Thanks. Understood. Let's reverse the question: If the CI server has done a good job to run through all tests, why do we bother to ask the developers to run the tests before push (assume it's successfully built locally). If the test failed, the CI server can email to the dev team and follow up action would be taken. – Frankie Hung Jul 06 '18 at 10:13
  • You are welcome :) If the other developers update their working copy before that the fix was pushed, there are stuck with an incorrect source code. It is frustrating (is it my regression or it is one of my peers ?) And it may also cost time : suppose that the problem takes 1 day to be corrected. What the other developers should do during this time ? – davidxxx Jul 06 '18 at 10:47
  • Hi @davidxxx, thanks for the feedback above. I am thinking of another scenario. How do we apply CI practice in test-driven development (TDD) where the test are initial failed. In the above discussion, the developers should fix tests before commit, which is impossible for TDD. – Frankie Hung Jul 12 '18 at 01:59
  • Hello Frankie Hung, sorry sometimes I read the comment and I forget to answer So to answer, in TDD you never commit a test that fails, you commit a test (or any code) only when all test pass (the famous green light). TDD is incremental : you implement progressively the implementation. If you use TDD and that you want to push tests early you should add a `//FIXME task` in your IDE for each test method to implement and additionally you could do it nicer by annotating the test methods with an annotation that triggers a warning at compile time. – davidxxx Aug 01 '18 at 20:14
  • Some references that can interest you : https://softwareengineering.stackexchange.com/questions/201743/tdd-and-version-control and https://stackoverflow.com/questions/1752607/how-to-intentionally-cause-a-custom-java-compiler-warning-message – davidxxx Aug 01 '18 at 20:14
1

Yes, they shall be run twice. Because if some developers don't, then they run never. Developers should run the tests locally to make sure that their code works correctly.

Your CI system is however your reference, so there's no chance of one person arguing that it "works on my machine", but fails for others. Looking forward to continuous delivery, knowing this state on the CI/CD system becomes even more important.

You might hope that always and forever, every commit has been tested successfully locally (and all workstations are the same and identical with production systems...), but hope is a bad strategy.

StephenKing
  • 36,187
  • 11
  • 83
  • 112