14

We have a backend test suite written with XCTest. The suite runs great in Xcode, but for various reasons it would be nice for us if we could also run the suite in an iOS app. Is that possible? I don’t mind writing some glue code for it, but as it is I can’t even import the XCTest framework in a non-testing target:

SomeController.swift:2:8: Cannot load underlying module for 'XCTest'
zoul
  • 102,279
  • 44
  • 260
  • 354
  • What are you trying to achieve? – Jon Reid May 03 '17 at 04:03
  • It would be a simple and convenient way for our engineers to run a backend test suite on demand from our debugging app. – zoul May 03 '17 at 04:38
  • That is rather unusual thing to do. I would run Jenkins (or any other CI) with this particular test on demand not the app itself. – Michał Myśliwiec May 04 '17 at 13:35
  • To clarify, the tests are not a part of the app being tested. They test the server component. I also think running them on a CI server is the correct solution, but until we get a CI server I would like to have a simple way of running the tests without a developer machine. – zoul May 04 '17 at 13:40
  • OK, then it shouldn't be XCTest test but rather some custom code and custom UI to show the results. – Michał Myśliwiec May 04 '17 at 14:00
  • Then I could not run the tests as XCTest, with the nice support in Xcode. Is there something magical about XCTest that prevents it from running in an app? – zoul May 04 '17 at 17:37
  • Yes, well. But without this nice Xcode support how will you gather/visualise test results within iOS app? Hmm, maybe you could try to mock XCTest classes and XCTAsserts methods. This way you could use the same code with XCTest and within iOS app. I don't know XCTest internals but I guess it's to tightly bound with macOS and Xcode to just import it in iOS app. – Michał Myśliwiec May 04 '17 at 20:40

1 Answers1

19

It is possible! The key is to get a copy of the XCTest framework linking and then use the public XCTest APIs to run your tests. This will give you the same console output you see when running the tests via Xcode. (Wiring up your own, custom reporter looks doable using the public APIs, but asking how to do so would make a good question in itself - not many people use the XCTest APIs, because Xcode does the dirty work for us.)

Copy In XCTest.framework

Rather than unravel whatever is breaking the module loading, you can copy the XCTest.framework bundle into your project from the platform frameworks for your target platform:

mkdir Frameworks && mkdir Frameworks/iphonesimulator
PLATFORMS=/Applications/Xcode.app/Contents/Developer/Platforms
FRAMEWORKS=Developer/Library/Frameworks
cp -Rp \
    "$PLATFORMS/iPhoneSimulator.platform/$FRAMEWORKS/XCTest.framework" \
    Frameworks/iphonesimulator/

Link Against Your Own XCTest

Then, pop open the target editor for your main app target, and drag and drop your copy from Finder to the list of "Linked Frameworks and Libraries".

Build Your Tests Into the Main App Target

Now, go to your test files, pop open the File Inspector, and tick the box next to your main app target for those files. Now you're building the test files as part of your main app, which puts all your XCTestCase subclasses into the binary.

Run Those Tests

Lastly, wire up a button to "Run Tests" to an action like this:

import UIKit
import XCTest

class ViewController: UIViewController {
    @IBAction func runTestsAction() {
        print("running tests!")
        let suite = XCTestSuite.default()
        for test in suite.tests {
            test.run()
        }
    }
}

When you tap the button, you'll see this in the console:

running tests!
Test Suite 'RunTestsInApp.app' started at 2017-05-15 11:42:57.823
Test Suite 'RunTestsInAppTests' started at 2017-05-15 11:42:57.825
Test Case '-[RunTestsInApp.RunTestsInAppTests testExample]' started.
2017-05-15 11:42:57.825 RunTestsInApp[2956:8530580] testExample()
Test Case '-[RunTestsInApp.RunTestsInAppTests testExample]' passed (0.001 seconds).
Test Case '-[RunTestsInApp.RunTestsInAppTests testPerformanceExample]' started.
/Users/jeremy/Workpad/RunTestsInApp/RunTestsInAppTests/RunTestsInAppTests.swift:34: Test Case '-[RunTestsInApp.RunTestsInAppTests testPerformanceExample]' measured [Time, seconds] average: 0.000, relative standard deviation: 122.966%, values: [0.000002, 0.000001, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[RunTestsInApp.RunTestsInAppTests testPerformanceExample]' passed (0.255 seconds).
Test Suite 'RunTestsInAppTests' passed at 2017-05-15 11:42:58.081.
     Executed 2 tests, with 0 failures (0 unexpected) in 0.256 (0.257) seconds
Test Suite 'RunTestsInApp.app' passed at 2017-05-15 11:42:58.081.
     Executed 2 tests, with 0 failures (0 unexpected) in 0.256 (0.258) seconds
zoul
  • 102,279
  • 44
  • 260
  • 354
Jeremy W. Sherman
  • 35,901
  • 5
  • 77
  • 111
  • This looks great, can’t wait to test it. Thank you! – zoul May 16 '17 at 02:53
  • :0: error: -[MyProjectTests testExample] : failed: caught "NSInternalInconsistencyException", "No target application path specified via test configuration: (null)" – laoyur May 16 '18 at 10:05
  • 1
    This works for me, I described a few tweaks in this answer here if anyone hits issues: https://stackoverflow.com/questions/54147582/how-to-link-xctest-dependency-to-production-main-target/54209379#54209379 – brthornbury Jan 16 '19 at 01:54
  • 1
    On Xcode 11.3.1 we also need to copy and link `XCTAutomationSupport.framework`. We can add to the script: `cp -Rp \ "$PLATFORMS/iPhoneSimulator.platform/$FRAMEWORKS/../PrivateFrameworks/XCTAutomationSupport.framework" \ Frameworks/iphonesimulator/` And drag and drop `XCTAutomationSupport.framework` too. – kanobius Mar 31 '20 at 20:53
  • On Xcode 11.5, we also need to copy and link libXCTestSwiftSupport.dylib, available in /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/lib folder. – Ivo Leko Jun 16 '20 at 14:32
  • In Xcode 14 this seems no longer possible. After following the last hint of @IvoLeko I had to also add XCTestCore.framework, and then I get a linker warning: "your binary is not an allowed client of XCTestCore". :( – Kristof Van Landschoot Dec 19 '22 at 16:14