DeploymentItem and TestCleanup conflict in unit tests?

Question

I have an application that has many unit tests in many classes. Many of the tests have DeploymentItem attributes to provide required test data:

[TestMethod]
[DeploymentItem("UnitTesting\testdata1.xml","mytestdata")]
public void Test1(){
    /*test*/
}

[TestMethod]
[DeploymentItem("UnitTesting\testdata2.xml","mytestdata")]
public void Test1(){
    /*test*/
}

When tests are run individually, they pass. When all are run at once (For example, when I select "Run all tests in the current context"), some tests fail, because the DeploymentItems left behind by other tests cause the tests to grab the wrong data. (Or, a test incorrectly use the files meant for another test that hasn't run yet.)

I discovered the [TestCleanup] and [ClassCleanup] attributes, which seem like the would help. I added this:

[TestCleanup]
public void CleanUp(){
    if(Directory.Exists("mytestdata"))
        Directory.Delete("mytestdata", true);
}

The trouble is, this runs after every test method, and it seems that it will delete DeploymentItems for tests that have not run yet. [ClassCleanup] would prevent this, but unfortunately, it would not run often enough to prevent the original issue.

From the MSDN documentation, it seems that DeploymentItem only guarantees that the files will be there before the test executes, but it is not more specific than that. I think I am seeing the following problem:

Deployment Item for test executes
(other stuff happens?)
Test cleanup from previous test executes
Next test executes
Test fails because files are gone

Does anyone know the execution order of the different test attributes? I've been searching but I haven't found much.

I have thought about having each deployment item use its own, unique folder for data, but this becomes difficult as there are hundreds of tests to go through.

*"some tests fail, because the DeploymentItems left behind by other tests cause the tests to grab the wrong data"* - Why is that happening, are your unit tests modifying the deployment items during the test? If that's true you could simply create copies of the deployment items you need to modify in each test to preserve the original files... — Thomas C. G. de Vilhena, Jan 11 '13 at 21:31
Does each test use a different file but share the same folder? — Ryan Gates, Jan 11 '13 at 21:35

score 5 · Accepted Answer · edited May 23 '17 at 11:46

5

The order of the test attributes are as follows:

Methods marked with the AssemblyInitializeAttribute.
Methods marked with the ClassInitializeAttribute.
Methods marked with the TestInitializeAttribute.
Methods marked with the TestMethodAttribute.

Part of the problem is that Visual Studio runs tests in a non-deterministic order(by default but this can be changed) and multiple at a time. This means that you cannot delete the folder after each test.

In general, if you can avoid going to the disk for unit tests it will be much better. In general you don't want to have anything besides code that can break your tests.

edited May 23 '17 at 11:46

Community

1
1

answered Jan 11 '13 at 21:30

Ryan Gates

4,501
6
50
90

Yes, I agree about avoiding going to the disk. The trouble is that that I'm testing code the purpose of which is to read files, so there must necessarily be files. The files also represent large data sets, so I can't easily instantiate a comparable object in code, and the tests don't always have much value when run against small data sets. – SouthShoreAK Jan 11 '13 at 23:20
Still an issue I am working on, but you did answer my question about execution order, so I'll mark it. – SouthShoreAK Jan 22 '13 at 20:21
@SouthShoreAK You may want to look at [How to: Configure Test Deployment](http://msdn.microsoft.com/en-us/library/ms182475%28v=vs.110%29.aspx) to make sure that every path is configured correctly. – Ryan Gates Jan 23 '13 at 04:22
3

@SouthShoreAK Do you really want to test reading files from the file system, or do you want to test how your code handles data that comes from the file system? If it's the latter (and I expect it is), you want to *mock* the file system for your testing purposes. – Daniel Mann Aug 12 '13 at 02:57

score 1 · Answer 2 · answered Nov 23 '13 at 09:17

I had a similar problem. In few tests i need to delete a deployed item - all tests pass when run individually, but failed when run in a playlist. My solution is ugly, but simple: use a different folder for every test.

For example:

    [TestMethod]
    [DeploymentItem("Resources\\DSC06247.JPG", "D1")]
    public void TestImageUploadWithRemoval()
    {
        // Arrange
        myDeployedImagePath = Path.Combine(TestContext.DeploymentDirectory, "D1", "DSC06247.JPG");
        // Act ...
    }

    [TestMethod]
    [DeploymentItem("Resources\\DSC06247.JPG", "D2")]
    public void TestImageUploadWithoutRemoval()
    {
        // Arrange
        myDeployedImagePath = Path.Combine(TestContext.DeploymentDirectory, "D2", "DSC06247.JPG");
        // Act...
    }

DeploymentItem and TestCleanup conflict in unit tests?

2 Answers2