5

Consider this sample project: https://github.com/stremsdoerfer/TestPlan. This is just a Hello World that has two test plans: TestPlanUnit that only runs unit tests and TestPlanUI that only run UI tests.

Running the command below with Xcode 14.3, I'd expect TestPlanUI to be run, but instead only TestPlanUnit is run which is the default.

xcodebuild -scheme TestPlan -destination 'platform=iOS Simulator,id=<sim_id>' -testPlan TestPlanUI test-without-building

It works fine with Xcode 14.2. Using test instead of test-without-building also works fine.

Any thoughts?

streem
  • 9,044
  • 5
  • 30
  • 41
  • Having the same issue. Discovered that today. Wow that Apple doesn't fic this immediately. This is kind of a core cuntionality for heavy CI users. It means you cannot run different test plans anymore. Is there any way around this? – blackjacx Apr 24 '23 at 19:26
  • 1
    @blackjacx The only work-around I've found (aside from backing off to 14.2, which I'm seriously considering) is to create a copy of my Scheme and set the default test of that scheme. – gaige Apr 25 '23 at 11:04
  • Another approach to circumvent this problem is to switch back prior using testplans, e.g. by editing the scheme with a text editor. – mbi Jun 21 '23 at 07:23

4 Answers4

3

Having the same problem and reported an issue to Apple. The testPlan parameter seems to be ignored when test-without-building is passed.

stid
  • 31
  • 1
3

I also reported testPlan ignored when using xcodebuild test-without-building to Apple. Apple responded with: Potential fix identified - For a future OS update.

Marty Hoevenberg
  • 46
  • 3
2

Not a solution, but a potential work-around until Apple fixes it. If you make a copy of your Scheme and then set the default TestPlan for that Scheme to be the test you want to run, test-without-building will pay attention to the default test in the Scheme.

I've tested with fastlane and directly using xcodebuild and both worked this way. I don't love it, especially because my schemes are fairly complex. However, I did have success getting my tests to run.

Make sure your Run variants are all built. I ran into a problem where I'd copied my schemes a while back and one scheme diverged by the Run / Diagnostics settings for the Runtime Sanitization for a single TestPlan. As such, that scheme wanted a binary in the Variant-NoSanitizers when I ran my tests in CI with sanitization turned off.

gaige
  • 17,263
  • 6
  • 57
  • 68
1

Actually I got a working soloution that does not require any hacks

The first invocation of xcodebuild will create a xctestrun file:

xcrun xcodebuild build-for-testing -workspace <project_root>/<project_name>.xcworkspace -scheme <scheme> -destination 'platform=iOS Simulator,id=2253C0D7-C198-44AD-A98B-11301FD88F30'

This file will be located in the DerivedData folder and be named like this:

~/Library/Developer/Xcode/DerivedData/<build_folder>/Build/Products/<scheme>_<test_plan_name>_<device_type><ios-version>-<architecture>.xctestrun

An example would be MyScheme_Screenshots_iphonesimulator16.4-arm64.xctestrun

The second invocation of xcodebuild then uses this file to run the tests. It is not allowed to specify any of [-project, -scheme, -workspace] when using -xctestrun:

xcrun xcodebuild test-without-building -destination 'platform=iOS Simulator,id=2253C0D7-C198-44AD-A98B-11301FD88F30' -xctestrun ~/Library/Developer/Xcode/DerivedData/<build_folder>/Build/Products/<scheme>_<test_plan_name>_<device_type><ios-version>-<architecture>.xctestrun

Hope this helps you folks

PS: Here you can find the Pull Request for my command line tool Snap that integrates this fix. Snap creates screenshots using Xcode TestPlans. Super useful for CI

blackjacx
  • 9,011
  • 7
  • 45
  • 56