2

I have a C# Visual Studio 2017 solution with multiple projects and test projects, a vNext build pipeline with VS Test task. The "Run only impacted tests" option is enabled. Clean options are unchecked.

I start a build with no changesets (no code changes). In the test agent's workspace, I can verify that there is no rebuild occurring: the project references, the Copy Local's, pre/post build events, Copy if Newer's are all good. No assemblies are replaced after the build.

Tests from some projects are not run (which is correct), but tests from some projects are always run.

The project properties of these test projects seem identical.

Does anybody have a suggestion on where to look?

Kemal Taşkın
  • 545
  • 4
  • 18
  • Are there any failed tests in previous testing? or some new files generated during last build that effect the current tests... – Andy Li-MSFT Sep 13 '18 at 07:08
  • @AndyLi-MSFT, there are no failing tests, no new files, no new assemblies.. – Kemal Taşkın Sep 13 '18 at 07:16
  • @KemalTaşkın - I have a VB Solution having 1 project and corresponding unit test project in it. TIA is enabled. But during code check-in, impacted unit test is not getting executed. If I add any new method and then write the corresponding unit test, then TIA is running newly unit test only. Could you please help me out here. – Piyush Jul 11 '19 at 09:58

2 Answers2

0

Please make sure no failed tests in previous build/test, and no new files generated during last build to effect current build/test.

Please refer to below links to know about what are the tests that get automatically selected and policies that can condition test selection for Test Impact Analysis .

Test Impact Analysis (TIA) will look at an incoming commit, and select the set of relevant tests – these will have 3 components

(1) The existing tests impacted by the incoming commit.

(2) Additionally, it will also select previously failing tests – if not, then over the course of several commits some earlier failing test case might just get lost … Therefore, TIA will keep track of tests that failed in the previous build and include them in the selection.

(3) It will also include newly added tests – what if your commit contains new tests? Such tests could uncover product bugs, right? So, TIA will select newly added tests as well.

TIA has:

(1) a robust test selection – this includes existing impacted tests, previously failing tests and newly added tests.

(2) safe fall back – for commits and scenarios that TIA cannot reason about, it will safely fall back to running all tests (you can see that mentioned in the logs). TIA is currently scoped to only managed code, and single box topology. So for e.g. if the code commit contains changes to HTML/CSS files, it cannot reason about them and will fall back to running all tests.

(3) configurable overrides – you can run “all” tests at a configured periodicity.

Andy Li-MSFT
  • 28,712
  • 2
  • 33
  • 55
  • Thank you for your answer. I've read those articles of course and I have an idea how TIA works. After days of inspection I found the cause of my issue and I'll write it as answer. – Kemal Taşkın Sep 13 '18 at 09:27
0

After days of inspection I've found the cause of this issue. I'm sharing this so maybe it helps others.

VS Test task (actually the vstest.console.exe in it) publishes results to a trx file and a number of text files that contains the dependencies of the tests.

1) If you have tests having the same (case insensitive) name, this publishing works unreliably. As it turned out, we had tests like this (i.e. TestThisWhenStringIsTrue and TestThisWhenStringIsTRUE).

2) Full names of the test methods (..) should not exceed 256 characters. When this is the case, the publishing works again unreliably.

After refactoring the tests the TIA started working. I hope this helps.

Note: Before coming to this step, you first need to have your incremental build running of course.

Kemal Taşkın
  • 545
  • 4
  • 18