C# Unit Tests, how do they behave

Question

I'm currently having a very strange issue with Unit Tests in my Visual Studio project. I have written a LogManager that takes various parameters, including a level to determine if the LogEntry should be written. Now I have 2 unit tests that tests them. Both including another value of the log entries.

Here's my first class:

/// <summary>
///     Provides the <see cref="ArrangeActAssert" /> in which the tests will run.
/// </summary>
public abstract class Context : ArrangeActAssert
{
    #region Constructors

    /// <summary>
    ///     Creates a new instance of the <see cref="Context" />.
    /// </summary>
    protected Context()
    {
        MapperConfig.RegisterMappings();

        var settingsRepository = new Repository<Setting>();
        settingsRepository.Add(new Setting { Key = "LOGLEVEL", Value = "DEBUG" });

        // Mock the IUnitOfWork.
        var uow = new Mock<IUnitOfWork>();
        uow.SetupGet(x => x.LogRepository).Returns(new Repository<Log>());
        uow.SetupGet(x => x.SettingRepository).Returns(settingsRepository);

        unitOfWork = uow.Object;
    }

    #endregion

    #region Properties

    /// <summary>
    /// The <see cref="IUnitOfWork"/> which is used to access the data.
    /// </summary>
    protected IUnitOfWork unitOfWork;

    #endregion Properties

    #region Methods

    #endregion Methods
}

/// <summary>
/// Test the bahviour when rwiting a new log.
/// </summary>
[TestClass]
public class when_writing_a_trace_log : Context
{
    #region Context Members

    /// <summary>
    /// Write a log.
    /// </summary>
    protected override void Act()
    {
        var httpApplicationMock = new Mock<IHttpApplication>();
        httpApplicationMock.SetupGet(x => x.IP).Returns("127.0.0.1");
        httpApplicationMock.SetupGet(x => x.RequestIdentifier).Returns(Guid.NewGuid().ToString().ToUpper());
        httpApplicationMock.SetupGet(x => x.UserIdentifier).Returns(Guid.NewGuid().ToString().ToUpper());

        LogManager.Write(httpApplicationMock.Object, unitOfWork, LogLevel.Trace, "Unit test", "This message is being writted using the LogManager.");
    }

    #endregion

    #region Methods

    /// <summary>
    /// Checks if the repository of the logs does contain an entry.
    /// </summary>
    [TestMethod]
    public void then_the_repository_should_contain_another_log_entry()
    {
        Assert.AreEqual(0, unitOfWork.LogRepository.GetAll().Count(), "The repository containing the logs does either not contain an entry or has more than a single entry.");
    }

    #endregion
}

In the example above, this class has a single method to test, but in fact there are 5. 1 for writing at each log level which can be 'Trace', 'Debug', Information', 'Warning', 'Error' or 'Critical'.

The purpose of this class is the following:

Is a log entry saved at a particular level when the level in the database is set to 'TRACE'.

Now, I have a copy of this class that has the same methods, to only difference is the context constructor which holds a LOGLEVEL of 'DEBUG' in the settings repository.

When I run all the unit tests right now, one test is failing (which makes no sense to me because it was working before and I haven't changed the code - apart from adding new classes with unit tests). When I debug the failed unit test, everything is correct.

And finally, here's the class under test:

public static class LogManager
{
    #region Methods

    /// <summary>
    ///     Writes a log message.
    /// </summary>
    /// <param name="application">The <see cref="IHttpApplication"/> which is needed to write a log entry.</param>
    /// <param name="unitOfWork">The <see cref="IUnitOfWork" /> used to save the message.</param>
    /// <param name="level">The <see cref="LogLevel" /> that the message should have.</param>
    /// <param name="title">The tiel of that the message should have.</param>
    /// <param name="message">The message to write.</param>
    public static void Write(IHttpApplication application, IUnitOfWork unitOfWork, LogLevel level, string title, string message, params AdditionalProperty[] properties)
    {
        if (CanLogOnLevel(unitOfWork, level))
        {
            var entry = new LogViewModel
            {
                Level = (Data.Enumerations.LogLevel)level,
                Client = application.IP,
                UserIdentifier = application.UserIdentifier,
                RequestIdentifier = application.RequestIdentifier,
                Title = title,
                Message = message,
                AdditionalProperties = new AdditionalProperties() { Properties = properties.ToList() }
            };

            unitOfWork.LogRepository.Add(Mapper.Map<Log>(entry));
        }
    }

    /// <summary>
    ///     Check if the log should be saved, depending on the <see cref="LogLevel" />.
    /// </summary>
    /// <param name="unitOfWork">The <see cref="IUnitOfWork" /> used to determine if the log should be written.</param>
    /// <param name="level">The <see cref="LogLevel" /> of the log.</param>
    /// <returns><see langword="true" /> when the log should be written, otherwise <see langword="false" />.</returns>
    private static bool CanLogOnLevel(IUnitOfWork unitOfWork, LogLevel level)
    {
        LogLevel lowestLogLevel = SingletonInitializer<SettingsManager>.GetInstance(unitOfWork).LogLevel;

        switch (lowestLogLevel)
        {
            case LogLevel.None:
                return false;
            case LogLevel.Trace:
                return level == LogLevel.Trace || level == LogLevel.Debug || level == LogLevel.Information ||
                       level == LogLevel.Warning || level == LogLevel.Error || level == LogLevel.Critical;
            case LogLevel.Debug:
                return level == LogLevel.Debug || level == LogLevel.Information || level == LogLevel.Warning ||
                       level == LogLevel.Error || level == LogLevel.Critical;
            case LogLevel.Information:
                return level == LogLevel.Information || level == LogLevel.Warning || level == LogLevel.Error ||
                       level == LogLevel.Critical;
            case LogLevel.Warning:
                return level == LogLevel.Warning || level == LogLevel.Error || level == LogLevel.Critical;
            case LogLevel.Error:
                return level == LogLevel.Error || level == LogLevel.Critical;
            case LogLevel.Critical:
                return level == LogLevel.Critical;
            default:
                return false;
        }
    }

    #endregion
}

Someone has a clue?

This unit test is passing when this unit test is runt:

/// <summary>
///     Provides the <see cref="ArrangeActAssert" /> in which the tests will run.
/// </summary>
public abstract class Context : ArrangeActAssert
{
    #region Constructors

    /// <summary>
    ///     Creates a new instance of the <see cref="Context" />.
    /// </summary>
    protected Context()
    {
        MapperConfig.RegisterMappings();

        var settingsRepository = new Repository<Setting>();
        settingsRepository.Add(new Setting { Key = "LOGLEVEL", Value = "DEBUG" });

        // Mock the IUnitOfWork.
        var uow = new Mock<IUnitOfWork>();
        uow.SetupGet(x => x.LogRepository).Returns(new Repository<Log>());
        uow.SetupGet(x => x.SettingRepository).Returns(settingsRepository);

        unitOfWork = uow.Object;
    }

    #endregion

    #region Properties

    /// <summary>
    /// The <see cref="IUnitOfWork"/> which is used to access the data.
    /// </summary>
    protected IUnitOfWork unitOfWork;

    #endregion Properties

    #region Methods

    #endregion Methods
}

/// <summary>
/// Test the bahviour when rwiting a new log.
/// </summary>
[TestClass]
public class when_writing_a_trace_log : Context
{
    #region Context Members

    /// <summary>
    /// Write a log.
    /// </summary>
    protected override void Act()
    {
        var httpApplicationMock = new Mock<IHttpApplication>();
        httpApplicationMock.SetupGet(x => x.IP).Returns("127.0.0.1");
        httpApplicationMock.SetupGet(x => x.RequestIdentifier).Returns(Guid.NewGuid().ToString().ToUpper());
        httpApplicationMock.SetupGet(x => x.UserIdentifier).Returns(Guid.NewGuid().ToString().ToUpper());

        LogManager.Write(httpApplicationMock.Object, unitOfWork, LogLevel.Trace, "Unit test", "This message is being writted using the LogManager.");
    }

    #endregion

    #region Methods

    /// <summary>
    /// Checks if the repository of the logs does contain an entry.
    /// </summary>
    [TestMethod]
    public void then_the_repository_should_contain_another_log_entry()
    {
        Assert.AreEqual(0, unitOfWork.LogRepository.GetAll().Count(), "The repository containing the logs does either not contain an entry or has more than a single entry.");
    }

    #endregion
}

Here's the ArrangeAct class:

/// <summary>
///     A base class for written in the BDD (behaviour driven development) that provide standard
///     methods to set up test actions and the "when" statements. "Then" is encapsulated by the
///     testmethods themselves.
/// </summary>
public abstract class ArrangeActAssert
{
    #region Methods

    /// <summary>
    ///     When overridden in a derived class, this method is used to perform interactions against
    ///     the system under test.
    /// </summary>
    /// <remarks>
    ///     This method is called automatticly after the <see cref="Arrange" /> method and before
    ///     each test method runs.
    /// </remarks>
    protected virtual void Act()
    {
    }

    /// <summary>
    ///     When overridden in a derived class, this method is used to set up the current state of
    ///     the specs context.
    /// </summary>
    /// <remarks>
    ///     This method is called automatticly before every test, before the <see cref="Act" /> method.
    /// </remarks>
    protected virtual void Arrange()
    {
    }

    /// <summary>
    ///     When overridden in a derived class, this method is used to reset the state of the system
    ///     after a test method has completed.
    /// </summary>
    /// <remarks>
    ///     This method is called automatticly after each testmethod has run.
    /// </remarks>
    protected virtual void Teardown()
    {
    }

    #endregion Methods

    #region MSTest integration

    [TestInitialize]
    public void MainSetup()
    {
        Arrange();
        Act();
    }

    [TestCleanup]
    public void MainTeardown()
    {
        Teardown();
    }

    #endregion MSTest integration
}

But when that same test in run in one single test run with other tests, the test is failing.

You haven't posted the class under test, so there isn't any way to determine what might be wrong. — Steve Mitcham, May 27 '14 at 12:57
I haven't because I tought it was irrelevant. I think it's MS test running test synchronously or something. I do behave it's the repository which is misbehving. Anyway, I've added the the class under test which can be found in the original post now. — Complexity, May 27 '14 at 12:59
I don't see anything that would be different in a synchronous vs parallel tests. Do you really expect the count to be zero in your assert? — Steve Mitcham, May 27 '14 at 13:02
Also, just for clarity, if your enum for log level is in ascending order you can just check for LogLevel >= lowestLogLevel instead of the switch. — Steve Mitcham, May 27 '14 at 13:04
I do expect that yet, and the problem lies in the copy of the class. Where in the SettingsRepository the level is set to trace, and then writing a trace message. The log repository don't have any entries then (only when I'm running all the tests). When I run the single test that fails, there is not any problem anymore and the test is running fine. — Complexity, May 27 '14 at 13:04
First, you shouldn't have 5 copies of the class or method. The logic should only exist in one place and then use parameters to configure how to write the files. I would look in the rest of your code for a discrepancy — Steve Mitcham, May 27 '14 at 13:08
To be honest, I don't think it's because of my files, but I do think it has something to do with the order how MSTest cleanup the state of tests. — Complexity, May 27 '14 at 13:13

weston · Accepted Answer · 2014-05-27T14:09:53.940

3

You're doing stuff in a constructor that should be done in a test initializer:

e.g. instead of:

protected Context()
{
    MapperConfig.RegisterMappings();

    var settingsRepository = new Repository<Setting>();
    settingsRepository.Add(new Setting { Key = "LOGLEVEL", Value = "DEBUG" });

    // Mock the IUnitOfWork.
    var uow = new Mock<IUnitOfWork>();
    uow.SetupGet(x => x.LogRepository).Returns(new Repository<Log>());
    uow.SetupGet(x => x.SettingRepository).Returns(settingsRepository);

    unitOfWork = uow.Object;
}

Do this:

protected Context()
{
    MapperConfig.RegisterMappings();
}

[TestInitialize]
protected void Setup()
{
    var settingsRepository = new Repository<Setting>();
    settingsRepository.Add(new Setting { Key = "LOGLEVEL", Value = "DEBUG" });

    // Mock the IUnitOfWork.
    var uow = new Mock<IUnitOfWork>();
    uow.SetupGet(x => x.LogRepository).Returns(new Repository<Log>());
    uow.SetupGet(x => x.SettingRepository).Returns(settingsRepository);

    unitOfWork = uow.Object;
}

So that before each test you get a clean mock unit of work.

A test run looks like this:

constructor
[ClassInitialize] methods.
for each [TestMethod]
  [TestInitlize] methods.
  [TestMethod]
  [TestCleanup] methods.
[ClassCleanup] methods.

The order of the TestMethods should never matter, i.e. you should never rely on the order. In your case if a test that adds a log entry to that unit of work runs before the test that checks that it's empty, then that test is going to fail. The solution is to always start with a clean unit of work.

Simple example of a bad test:

[TestClass]
public class Test
{
   private List<int> list;

   public Test()
   {
      list = new List<int>();
   }

   [TestMethod]
   public void can_add_to_list()
   {
      list.Add(10);
      Assert.areEqual(1, list.Count);
   }

   [TestMethod]
   public void can_add_two_to_list()
   {
      list.Add(10);
      list.Add(20);
      Assert.areEqual(2, list.Count);
   }
}

These tests will always work on their own, but when run together one of them will fail because the list is not created a fresh before each test.

edited May 27 '14 at 14:09

answered May 27 '14 at 13:17

weston

54,145
21
145
203

Thanks for your answer, but it isn't the solution for me. I'm sorry for that. – Complexity May 27 '14 at 13:19
I have tested that indeed, but it isn't working. My ArrangeActAssert base class has a TestInitialize that will run Arrange() first and then Act(), and I have moved the UnitOfWork to the Arrange() method in my Context – Complexity May 27 '14 at 13:29
Perhaps you can post that class. – weston May 27 '14 at 13:37
It has been added in my original post. – Complexity May 27 '14 at 13:38
I'm sorry but your answer was correct. With all the changes I made a stupid typo. But the initializer class was correct. But would you mind what the issue was because I've put it in my constructor? – Complexity May 27 '14 at 13:40
Well I have posted a simple example. It shows two tests that pass on their own, but one will fail when when run together. This is in essence what you had. – weston May 27 '14 at 14:10
Thanks. I still have an issue, but I believe that with the code that you posted, I might be able to resolve this :-) – Complexity May 27 '14 at 14:12
I've tried your code "areEqual" should be "AreEqual" in the first place, but when runt together both tests do pass. I do find it very strange but that's how it's behaving... – Complexity May 27 '14 at 14:14
I have finally found my issue. Inside my LogLevel class, I create a Singleton of a given class. Off course, when moving to the other unit test class that changes the properties for the Singleton, the Singleton does not see it because it still live. – Complexity May 27 '14 at 14:25
What test runner are you using? If it's one that runs each test in isolation, then it's possible that will pass. – weston May 27 '14 at 14:32
MSTest. So each test in separation. – Complexity May 27 '14 at 14:37
Hmm, looks like I am wrong. Suggest you add your own answer and tick it so I can delete this. It's all based on an assumption that the constructor executes once per test run. Which according to this and my tests is wrong: http://stackoverflow.com/questions/334515/do-you-use-testinitialize-or-the-test-class-constructor-to-prepare-each-test-an – weston May 27 '14 at 14:50
I'll add my answer this evening. Thx for the help anyway. – Complexity May 27 '14 at 15:10

C# Unit Tests, how do they behave

1 Answers1