Jenkins failed unit CanExecute test's methods nondeterministic

Question

We have a lot CanExecute tests for a various commands in our project. All tests passed properly when we use Visual Studio testing or AxoCover.

We tried to add some previous object initialization, before executing 'CanExecute' and sometimes it worked (or we thought that).

testedViewModel.Object.InEditMode = inEditMode;

I have a test:

[TestCase(true, true, TestName = "Command_InEditMode_CanExecute")]
[TestCase(false, false, TestName = "Command_NotInEditMode_CannotExecute")]
public void CommandCanExecute(bool inEditMode, bool expectedResult)
{
    var testedViewModel =
        new Mock<SomeViewModel>(inEditMode)
        {
            CallBase = true
        };

    testedViewModel.Setup(x => x.InEditMode).Returns(inEditMode);

    Assert.AreEqual(expectedResult, testedViewModel.Object.Command.CanExecute(null));
}

Sometimes (NOT ALWAYS) when Jenkins does the build and run unit tests some of can execute tests failed with message:

MESSAGE:
  Expected: True
  But was:  False

+++++++++++++++++++  
STACK TRACE:
   at Project.CommandCanExecute(Boolean inEditMode, Boolean expectedResult)

The problem is that is happening only on Jenkins and it's very nondeterministic.

EDIT:

Ok, one more thing to think about. Property InEditMode is placed in base parent class of SomeModelView.

And I merged code for you in the sample.

public BaseViewModel 
{
    public virtual bool InEditMode {get; set;}
}

public SomeViewModel : BaseViewModel
{
    public SomeViewModel () : base ()
    {

    }

    public ICommand Command { get; set; }

    public virtual void RegisterCommands()
    {
        Command = new RelayCommand(/*Do something*/, () => InEditMode);
    }
}

And we think that can be related, that object is thinking that is initialized before initialization of base class is done. But that is very hard to check this with a Jenkins.

SOLUTION

I've created an attribute class:

[AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Interface | AttributeTargets.Assembly, AllowMultiple = true)]
public class GarbageCollectorDisabler : Attribute, ITestAction
{
    public void BeforeTest(ITest test)
    {
        GC.TryStartNoGCRegion(2048 * 4096);
    }

    public void AfterTest(ITest test)
    {
        GC.EndNoGCRegion();
    }

    public ActionTargets Targets => ActionTargets.Test;
}

And then I can use for each 'CanExecute' test this attribute:

[GarbageCollectorDisabler]
[TestCase(TestName = "SomeTest_InEditMode_CanExecute")]
public void SomeTestCanExecute()
{}

Do you have racing conditions in your code? Sometimes we have a similar because our tests are executed in multiple threads — Michael Kemmerzell, Jun 04 '19 at 18:27
No, the code you see above it's almost completely the same in our project. We have very simple CanExecute sometimes, like above. And our Jenkins does not executed tests in multiple threads as far as I know. — Piotr Zieliński, Jun 05 '19 at 07:19
@PiotrZieliński The messaged says : `MESSAGE: Expected: True But was: False` the test you have showed inputs `false` every time for the expected result, so how could the expected be `true` in the message you showed us? — JohnChris, Jun 07 '19 at 10:51
I added second test case. CanExecute method is in the code. It's only that: () => InEditMode — Piotr Zieliński, Jun 07 '19 at 12:47
@PiotrZieliński How is it that `InEditMode` is able to be overridden in your example when it is not `virtual`? Provide a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) that can be used to reproduce the problem, allowing a better understanding of what is being asked. — Nkosi, Jun 07 '19 at 13:14
AFAIK you need a virtual or an interface or when you moq one - it'll error otherwise. Do you really have a property rather than method for canexecute? Because you could just set that. And.. What's that Object.Command all about? — Andy, Jun 08 '19 at 10:15
testedViewModel.Object is an instance of mocked object. And Command is a name of property in the class which contains RelayCommand object with an anonymous function. And you are right, InEditMode is a virtual type, forgot to add this in code above. — Piotr Zieliński, Jun 10 '19 at 11:30
Have you tried rewriting to **not** use parameters in method tests at all? — André Sanson, Jun 10 '19 at 16:00
How it can help here? And then had I write all test cases in different tests? It's not something I looking for I'm afraid. — Piotr Zieliński, Jun 11 '19 at 09:52
Can you try running with a single test to see if there's any issue ? If it runs every time with a single test, we can rule out the multi-threading / test-interleaving flows — Aditya, Jun 14 '19 at 09:21

score 1 · Accepted Answer · answered Jun 12 '19 at 20:18

1

Smells like a garbage collection issue to me. I don't see anything that jumps out at me in your sample, though the code sample is incomplete (where is RegisterCommands invoked?) so something crucial could be missing.

See the source for RelayCommand.CanExecute(). It takes a weak reference to the action you pass in, and once that action is collected CanExecute will return false. See my answer here for an example of this happening.

I reiterate @Nkosi's comment, create a minimal example rather than showing us bits and pieces.

answered Jun 12 '19 at 20:18

Patrick Quirk

23,334
2
57
88

Yes, it was a garbage collector issue! I will add solution to my original post. – Piotr Zieliński Aug 19 '19 at 13:12
2

@PiotrZieliński Not a huge fan of that solution. Your code has a bug (some object is not being retained correctly) and at some point you'll hit it in a non-test environment. Disabling the GC during test will just mask the bug. – Patrick Quirk Aug 19 '19 at 13:28

Jenkins failed unit CanExecute test's methods nondeterministic

SOLUTION

1 Answers1