0

I'd like to batch UI tests (right now, I guess unit tests would be useful some time)

Neither Xcode UI tests nor "fastlane scan" seem to have any inherent support for this as of now. Is there a good and simple approach?

Situation

I need to test a certain UI test N times.

Real world situation

I need to test a certain UI test 100 times and get an output at the end of batch:

  • how many iterations succeeded / failed
  • any logs of the failed iterations

What I have tried (and am doing as of now)

In a fastfile, fastlane scan could be used like this to batch. However the output files all end up in separate directories and it would be quite (?) some work aggregating all these results. It's not like scan returns a boolean or anything denoting the success status of the test? (Also actually scan does likely not support running a certain test, it runs them all)

100.times do |index|
    puts "Running test iteration #{index}..."
    scan(scheme: schemefortesting,
        output_directory: "fastlane/tests/test_output_#{index}",
        destination: 'name=Myrealworldiphonename'
    )
end
Jonny
  • 15,955
  • 18
  • 111
  • 232

1 Answers1

1

As far as I know there isn't support for this. I'd imagine the best way to do it would be to make a method to execute the test then have the test case run it many times, keeping track of the results overall and logging to a console. Something like:

func runTheTest() -> Bool {
    app.launch()
    // Run your test
    let passing = // Make your asserts
    return passing
}

func testManyTimes() {
    var allTestsPassing = true
    for n in 0...100 {
        let thisLoopPassing = runTheTest()
        print("Loop \(n) returned \(thisLoopPassing)")
        allTestsPassing = allTestsPassing && thisLoopPassing
    }
    XCTAssertTrue(allTestsPassing)
}

Idea taken partly from Automatically Running a Test Case Many Times in Xcode .

alannichols
  • 1,496
  • 1
  • 10
  • 20
  • It is another way I agree, and I also tried it out at first. And it might work for some particular tests. I needed to test something that would require a complete fresh launch each time, I.e. starting a new process each time. However I do not think there is a way to completely shut down the process and launch it again during the course of one test (?). Running the setup and cleanup methods over and over just was not enough in my case. – Jonny May 31 '18 at 14:37
  • You should be able to use app.launch() and app.terminate() to start and stop the app. But I'm not sure about uninstalling. – alannichols Jun 05 '18 at 16:16