38

Is it possible to run individual test cases, or individual test suites, from an iOS app test target, instead of all the test cases, from a command line interface?

You can run tests from command line with xcodebuild, out of the box. When you do so, you run all of the test cases contained in the test target you've selected.

You can also do so with scan from Fastlane, though I believe you're restricted to running all of the tests of the build scheme you select (as above), so it's not different from xcodebuild.

You can run specific tests with xctool from Facebook, but it doesn't use xcodebuild, and is restricted to running on simulators only, not actual iOS test devices.

I found a reference to running the xctest command line utility directly, but it seems to be an undocumented feature and targets DerivedData. This is complicated by the fact that UI Tests, have their *xctest files in a separate XCTRunner bundle.

Community
  • 1
  • 1
Apophenia Overload
  • 2,485
  • 3
  • 28
  • 46
  • should you have an example with xctool to run specific UI Tests? – emoleumassi Feb 03 '16 at 13:58
  • @emoleumassi xctool has the "-only" option to let you specify individual test files (suites) and test methods (cases) to run at a time. It works for unit tests and I assume it does the same for UI tests. I haven't tried it myself, though, because I require a testing solution to run on a real device, not on simulators only, so xctool is not a solution. – Apophenia Overload Feb 03 '16 at 21:35
  • 1
    xctool doesn't support UI test, see this answer: http://stackoverflow.com/questions/35181126/xctool-error-build-tests-is-not-a-testing-target-in-this-scheme/35196209#35196209 – emoleumassi Feb 04 '16 at 17:35
  • 1
    Thanks for the heads up. I would have thought that it's possible to trigger Xcode UI Tests the same way we run Unit Tests, but I guess I'm wrong. – Apophenia Overload Feb 04 '16 at 17:37

7 Answers7

103

It is now possible with Xcode 8 using the -only-testing parameter with xcodebuild:

xcodebuild test -workspace <path>
                -scheme <name>
                -destination <specifier>
                -only-testing:TestBundle/TestSuite/TestCase

enter image description here

Tejasvi Manmatha
  • 564
  • 7
  • 12
emoleumassi
  • 4,881
  • 13
  • 67
  • 93
  • 3
    Perfect. Thanks for the screenshot. It's also worth mentioning, as it was news to me, that we've also got new xcodebuild actions to split out the components of the original `test` action - namely `build-for-testing` and `test-without-building`. The latter being handy if you want to run a test multiple times without the overhead of building each time. Also great for aligning with Maven lifecycle phases if you need that. – Quintin Willison Dec 20 '16 at 18:19
  • 2
    `xcodebuild test -workspace -scheme -destination -only-testing:TestBundle/TestSuite/TestCase` – plosco Sep 29 '17 at 00:17
  • Is it possible to filter using asterisks? – ReDetection Jul 04 '18 at 10:12
27

You can edit the scheme to run only specific tests. Select the scheme, then edit scheme. In the appearing window, select the Test phase and disable/enable individual tests.

enter image description here

You can also add schemes to run subsets of tests. When running the tests from command line you can specify the scheme to use for the test (at least in fastlane).

dasdom
  • 13,975
  • 2
  • 47
  • 58
  • for 100 subsets of tests, you will have 100 schemes. No a good simulator. I prefer the other one with environment variable. – emoleumassi Feb 22 '16 at 13:18
19

Run an individual XC test

To run an individual test case you can use -only-testing

-only-testing Pattern

-only-testing:<target>/<class_name>/<test_name>

xcodebuild Pattern

xcodebuild test 
-workspace "<name>.xcworkspace"  
-scheme "<name>" 
-destination '<options>' 
-only-testing "<test_case>"

Example

//for example(several test cases)
xcodebuild test 
-workspace "MyApp.xcworkspace"
-scheme "MyAppTest" 
-destination 'platform=iOS Simulator,name=iPhone 11 Pro Max,OS=13.5' 
-only-testing "MyAppTest/TestClass/foo"
-only-testing "MyAppTest/TestClass/foo2"

For example if Test Navigator looks like

the parameter will have the following type

-only-testing:F49088168M_OBDIITests/HexUtilityTests/testHexToBinStringFormat

If you want to add an additional test case you can add one more -only-testing

Also you can skip a test using: -skip-testing

Test results you can find in Derived Data[About]

<derived_data>/<project_name>-dzqvyqfphypgrrdauxiyuhxkfxmg/Logs/Test/Test-<target_name>-<date>.xcresult

[Xcode screenshot]

yoAlex5
  • 29,217
  • 8
  • 193
  • 205
6

I was in similar situation as you and have built a python script that triggers the set of test case/s that i want. Its a little bit elaborate process but works for me and has been very useful over time in implementing DataProvider methods, Rerunning of failed test cases and other customizations I required.

Some relevant steps for what you want to achieve.

  1. Override testInvocations method present in XCTestCase to do below steps
    • In this method I read Environment Variable XXXX.
    • This environment variable is basically a comma separated test case method names.
    • Now create NSInvocations for each test method that you want to trigger.
    • Return array of Invocations.
  2. How to pass Environment Variable?
    • In scheme add an environment variable named XXXX.
    • Scheme files are standard xml files, write a script that modifies the scheme file to contain the Comma separated values in environment variable.

If you require more info add a comment I will reply to it.

Naveen Kumar S R
  • 129
  • 5
3

To run an individual test or test class you can click the diamond next to it in the gutter. This is right next to where line numbers appear if you have them turned on.

In this screenshot my mouse is hovering over the diamond. Notice how it has changed to a little play arrow indicating it will be run.

Hovering Over the Test Diamond

You can then re-execute the most recently run test(s) with ⌃⌥⌘ G.

As far as I know this cannot be done via the xcodebuild.

Joe Masilotti
  • 16,815
  • 6
  • 77
  • 87
  • 2
    I've clarified the original question. My main intent was asking if individual test cases can be triggered from command line, so there could be continuous integration that only calls a subset of tests instead of the entire target. – Apophenia Overload Feb 03 '16 at 17:28
  • See @dasdom's answer. There is some discussion going on in the Swift mailing list that is aiming to add this exact feature. – Joe Masilotti Feb 03 '16 at 19:22
  • While you didn't answer the OP's question, this was exactly _my_ question, I really appreciate your answer! – Lord Zsolt Jan 23 '18 at 14:25
0

May be this helps. I was ignorant because I didn't know that you can disable some particular tests in Test navigator by right-clicking them.

Then you run xcodebuild test -scheme <name> as usual, and all the disabled tests will be ignored. Worked with Fastlane for me.

Xcode Test navigator

kelin
  • 11,323
  • 6
  • 67
  • 104
0

If you don't have .xcworkspace. Use -project instead.

xcodebuild test \
-project "name.xcodeproj" \
-scheme "scheme name" \
-destination 'platform=iOS Simulator,name=iPhone 14 Pro,OS=16.4' \
-only-testing:<test target name>/<test class name>/<test function name>
Zhou Haibo
  • 1,681
  • 1
  • 12
  • 32