0

So I'm creating an NUnit project where each test:

  • runs a new Process from System.Diagnostics
  • uses cmd.exe to call lli.exe with an LLVM code file as an argument
  • checks exit code and output of this command

Everything goes well and tests are passed when they are run separately, one by one from the Test Explorer. However, when I try to run more than one test in a run, the problem occurs. Here is my code:

        [Theory]
        [TestCase("TestFile1", "")]
        [TestCase("TestFile2", "")]
        [TestCase("TestFile3", "0X17FFAA")]
        public void TestValidProgram(string programPath, string expectedOutput = "")
        {
            Compiler.Compile(programPath); // here {programPath}.ll file is created
            
            string result;
            ProcessStartInfo startInfo = new ProcessStartInfo
            {
                FileName = "cmd.exe",
                Arguments = $"/C {lliPath} {programPath}.ll",
                UseShellExecute = false,
                RedirectStandardOutput = true,
                RedirectStandardError = true,
                CreateNoWindow = true
            };
            using (Process proc = new Process())
            {
                proc.StartInfo = startInfo;
                proc.Start();
                proc.WaitForExit();
                result = proc.StandardOutput.ReadToEnd();
                Console.WriteLine($"res: {result}");
                Assert.AreEqual(0, proc.ExitCode); // stopping here
                proc.Close();
            }

            Assert.AreEqual(expectedOutput, result);
        }

When I run all 3 tests in a run, only the first, TestFile1 passes and the rest stop at this assertion:
Assert.AreEqual(0, proc.ExitCode); with proc.ExitCode equal 1. Also, the result string is empty (as it shouldn't be in case of TestFile3).

This is not the case with parallel run, I run them sequentially. Also, adding the [NonParallelizable] attribute doesn't change anything. Files created by Compiler are created properly and can be run successfully with lli.exe.

I use VSCode 2019 16.10.2. The project is run on .Net Framework 4.8 and NUnit 3.13.2. (I know I could use .net core or .net5 but this is the requirement for the project).
I'm running out of ideas, need help! :)

daveedoo
  • 1
  • 1

1 Answers1

0

The method under [SetUp] attribute will execute each time before executing every test case/method.

Ensure all previous processes were terminated before executing another test case in this method.

Gowthaman
  • 40
  • 5
  • Shouldn't it be terminated? I mean, I use `proc.WaitForExit();` and I dispose the `proc`. I just checked for the `proc.HasExited` property - it is set to `true` after exiting, as can be expected. – daveedoo Jun 24 '21 at 18:46
  • Guessing issue with `ProcessStartInfo`. Try using this in `[SetUp]` method and use `startInfo.Arguments` alone in test method. – Gowthaman Jun 24 '21 at 18:59
  • @Gowthaman The context of a setup method is identical to that of the test method. However, it does look as if a TearDown method may be needed to clean up after each test. I too would be suspicious of the Compile step. – Charlie Jun 24 '21 at 21:36