Can I run an XCTest suite multiple times?

Question

Is it possible to have Xcode run your unit tests multiple times?

I had an issue in several unit tests that caused intermittent failures. Now that I think I've fixed it, my only option appears to mash ⌘ + U until I'm 95% confident the bug is gone.

I know other unit testing frameworks make it quite easy to run a single test, test case, or test suite multiple times. Do we have this luxury in XCTest yet?

score 12 · Answer 1 · edited Apr 13 '21 at 11:37

12

For me it works in swift

override func invokeTest() {
    for time in 0...15 {
        print("this test is being invoked: \(time) times")
        super.invokeTest()
    }
}

edited Apr 13 '21 at 11:37

Parth

2,682
1
20
39

answered Dec 21 '18 at 07:49

Sultan Ali

2,497
28
25

This worked great for me in Xcode 10, but now in 11 not having any luck with it. – David H Feb 02 '20 at 20:28
well, i have tested it and it is working for me in xcode 11 as well – Sultan Ali Mar 04 '20 at 12:12
I tried with 11.3. On 11.4 beta now, will try again. I’ll try something trivial... – David H Mar 04 '20 at 12:28
This is working in Xcode 14.1 – Matea Apr 06 '23 at 13:32

score 7 · Answer 2 · answered Jun 28 '18 at 20:55

7

Try overriding invoke test: https://developer.apple.com/documentation/xctest/xctestcase/1496282-invoketest?language=objc

- (void)invokeTest
{
    for (int i=0; i<100; i++) {
        [super invokeTest];
    }
}

answered Jun 28 '18 at 20:55

electronix384128

6,625
11
45
67

score 4 · Accepted Answer · answered Jul 03 '15 at 08:00

4

It might help you to use

func testMultiple() {
    self.measureBlock() {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

This runs the code inside self.measureBlock() multiple times to measure the average time.

It is work to change the code, but you might want to know the execution time anyways.

This answer might be close enough to what you want and it is easy to do.

answered Jul 03 '15 at 08:00

Gerd Castan

6,275
3
44
89

1

That's true; I could wrap every test in a call to `measureBlock`, but does anyone know how many times the block is executed in `measureBlock`? – T Blank Jul 06 '15 at 16:35
The unit test tells you in the console when you run it. Example: measured [Time, seconds] average: 0.002, relative standard deviation: 154.282%, values: [0.011297, 0.001023, 0.000956, 0.000969, 0.000952, 0.000935, 0.000979, 0.000950, 0.000968, 0.001042], – Gerd Castan Jul 06 '15 at 18:51
What is bad in my answer is that the tests are not mixed randomly. If you have errors that occur from time to time, the sequence of tests might play a role. – Gerd Castan Jul 06 '15 at 18:55
I suppose I could just wrap each individual test in a `measureBlock` block, so they'd still run in a random order. Thanks for looking into this with me; I think this is the best we can do with XCTest. – T Blank Jul 08 '15 at 00:52
Does this call tearDown / setup after each run, I'm assuming not. – gran_profaci Jul 22 '16 at 22:44
Not while you're inside the method – Gerd Castan Jul 22 '16 at 22:52

score 3 · Answer 4 · edited Apr 25 '18 at 05:29

One alternative is to do this via the command line. You can run a single test using the -only-testing argument, and avoid building using test-without-building i.e. (new lines added for clarity)

for i in {1..10}; \
  do xcodebuild \
    test-without-building \
    -workspace MyApp.xcworkspace \
    -scheme Debug \
    -destination 'platform=iOS Simulator,OS=11.2,name=iPhone 8' \
    -only-testing:MyApp.Tests/TestFile/myTest;
done

score 3 · Answer 5 · answered Feb 01 '23 at 17:01

Xcode has a built-in way you can do this:

Right-click the test and select Run <test> Repeatedly...

That menu has a few configurations, one of them is Stop After where you can select Failure which will stop the execution if it failed during one of the repetitions.

You can also run a test class or the whole test suite repeatedly.

score 0 · Answer 6 · answered Feb 01 '17 at 21:32

Try using a for loop:

func testMultiple() {
    for _ in 0...100 {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

Note this doesn't work within a self.measureBlock(). You'll get an NSInternalConsistencyException: Cannot measure metrics while already measuring metrics

However, you can CALL this within a measureBlock():

func testMultiple() {
    for _ in 0...100 {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

func testPerformance() {
    self.measureBlock() {
        self.testMultiple()
    }
}

Xcode 8 runs the measureBlock code 10 times.

score 0 · Answer 7 · answered Feb 03 '20 at 13:41

I had used the invokeTest() override in the past (Xcode 10) with great success. But now in Xcode 11 its not working (for me at least). What I ended up doing was:

func test99Loop() {
    for i in 0..<LOOP_COUNT {
        if i > 0 { tearDown(); sleep(1); setUp() }
        test3NineUrls()
        do { tearDown(); sleep(1) }
        setUp()
        test6NineCombine()

        print("Finished Loop \(i)")
    }
}

I obviously use setup/teardown, and this is the proper way to do those multiple times (since the first and last are called by Xcode).

score 0 · Answer 8 · answered Jan 31 '23 at 01:33

You can now do this in Xcode

Edit the Test Plan. In the Test Navigator, at the top you should see "Test Plan: MyAppName (Default)". Tap on this and select "Edit Test Plan".

In the editor, select Configurations, Configuration 1

Under Test Execution you can set Test Repetition Mode and Maximum Test Repetitions.

Run the tests from the menu Product > Test

Can I run an XCTest suite multiple times?

8 Answers8

Linked