Automatically Running a Test Case Many Times in Xcode

Question

In Xcode, is there a way for me run a single test case n times automatically?

Reason for doing this is that some of my beta testers are encountering random crashes in my app. I see the crash logs in TestFlight, along with the stack trace, but I can't reproduce the crash.

The crash happens infrequently but when it does, it always happens when users are trying to create a DB record, which then gets uploaded to a server. The problem with the crash logs is that my code does not make an appearance in their stack traces (all UIKit & CoreFoundation stuff - and different each time).

My solution is to run the test for that part of the app 100s of times, with the exception breakpoint set, to try to trigger the bug in my dev environment. But I don't know how to do this automatically.

I know it's not exactly what you're looking for, but you could always just wrap the body of the test case in a loop. — Aaron Golden, Jul 12 '14 at 02:03
Hmm... I'll probably do this and just temporarily move my setup/teardown code into the actual test case and just run that one. — Shinigami, Jul 12 '14 at 08:11

score 19 · Answer 1 · answered Jun 14 '21 at 02:01

It took 7 years, but as of Xcode 13, support for test repetition is now built-in.

From the Xcode 13 release notes:

Enable test repetition in your test plan, xcodebuild, or by running your test from the test diamond by Control-clicking and selecting Run Repeatedly to bring up the test repetition dialog.

score 18 · Answer 2 · edited Apr 13 '21 at 19:36

18

You can read my answer here.

Basically you need to override invokeTest method

override func invokeTest() {
    for time in 0...15 {
        print("this test is being invoked: \(time) times")
        super.invokeTest()
    }
}

edited Apr 13 '21 at 19:36

Parth

2,682
1
20
39

answered Jan 22 '19 at 12:20

Sultan Ali

2,497
28
25

1

Love it. This helps track down memory leaks!! You can just add it to one test class or to a base test class. How did you find this? – Alex Zavatone Jul 19 '19 at 18:14
1

Brilliant! With this in place, you can ALSO tap on one test in the XCTest, and have that one test done repeatedly. HUGE HELP! – David H Jul 26 '19 at 20:39

score 4 · Answer 3 · answered May 13 '17 at 22:11

In Xcode as such, no.

You can create an XCTestCase class that hooks into the test-running methods it inherits to return multiple runs, but that tends to be annoying and mostly undocumented.

It's probably easier to instead have a "meta-test" that calls out to your other test method repeatedly:

func testOnce() {}

func testManyTimes() {
    for _ in 0..<1000 { testOnce() }
}

You might need to call out to some per-test setup/teardown methods. You could perhaps work around that by instead making the loop body be something like:

let test = XCTestCase(selector: #selector(testOnce))
test.invokeTest()

This would lean on the XCTest machinery that your standard tests use, but it might gripe about not being wired into an XCTestCaseRun (or not).

Automatically Running a Test Case Many Times in Xcode

3 Answers3

Linked