Unable to run Xcode unit tests from the command line due to otest error

Question

I'm trying to run unit tests from the command line but they fail while they run perfectly fine in Xcode 4.6. I can't figure out what's the cause of failure:

Here's the command I'm using:

xcodebuild -sdk iphonesimulator -project myproject.xcodeproj 
    -scheme 'MyProjectApplicationTests' 
    -configuration Debug clean build 
    RUN_UNIT_TEST_WITH_IOS_SIM=YES TEST_AFTER_BUILD=YES TEST_HOST=''

I have no usage of NSURL or to 'initFileURLWithPath' in my tests. Here's the error:

/Applications/Xcode.app/Contents/Developer/Tools/RunPlatformUnitTests.include:412: note: Started tests for architectures 'i386'
Run unit tests for architecture 'i386' (GC OFF)
/Applications/Xcode.app/Contents/Developer/Tools/RunPlatformUnitTests.include:419: note: Running tests for architecture 'i386' (GC OFF)
2013-06-04 22:09:57.605 otest[85321:707] Unknown Device Type. Using UIUserInterfaceIdiomPhone based on screen size
2013-06-04 22:09:57.624 otest[85321:707] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[NSURL initFileURLWithPath:]: nil string parameter'
*** First throw call stack:
(0xa05012 0x71ce7e 0xa04deb 0x2a9b1 0x2a93b 0x4d3c5c9 0x71d7cf 0x724a0d 0x71baeb 0x71be22 0x72e0e1 0x2010879a 0x20106ef5 0x20107124 0x20107196 0x2010624c 0x201063da 0x7305c8 0x2342 0x25ef 0x268c 0x2001 0x1f71)
libc++abi.dylib: terminate called throwing an exception
/Applications/Xcode.app/Contents/Developer/Tools/RunPlatformUnitTests.include: line 415: 85321 Abort trap: 6           "${THIN_TEST_RIG}" "${OTHER_TEST_FLAGS}" "${TEST_BUNDLE_PATH}"
/Applications/Xcode.app/Contents/Developer/Tools/RunPlatformUnitTests.include:451: error: Test rig '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator6.1.sdk/Developer/usr/bin/otest' exited abnormally with code 134 (it may have crashed).

If there is a way to even print out the callstack (in English), I could maybe figure out where the error is from.

Answer 1

The issue is that xcodebuild does not natively support application tests in the simulator. To run applications tests in the simulator from the command line you need an open source utility ios-sim and a tweek to your tests target run script.

1) Install ios-sim https://github.com/phonegap/ios-sim

$ curl -L https://github.com/phonegap/ios-sim/zipball/1.9.0 -o ios-sim-1.9.0.zip
$ unzip ios-sim-1.9.0.zip 
$ cd phonegap-ios-sim-538ef1a/
$ sudo rake install prefix=/usr/local/

2) Edit your tests target run script. This script is from Atlassian's fabulous documentation. I love Atlassian makers of JIRA, Stash, Confluence, and my new best friend SourceTree. Full disclosure they acquired SourceTree and it's developer then released it for FREE.

Replace existing:

# Run the unit tests in this test bundle.
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"

With this: (Source https://confluence.atlassian.com/display/BAMBOO/Xcode)

if [ "$RUN_UNIT_TEST_WITH_IOS_SIM" = "YES" ]; then
    test_bundle_path="$BUILT_PRODUCTS_DIR/$PRODUCT_NAME.$WRAPPER_EXTENSION"
    ios-sim launch "$(dirname "$TEST_HOST")" --setenv DYLD_INSERT_LIBRARIES=/../../Library/PrivateFrameworks/IDEBundleInjection.framework/IDEBundleInjection --setenv XCInjectBundle="$test_bundle_path" --setenv XCInjectBundleInto="$TEST_HOST" --args -SenTest All "$test_bundle_path"
    echo "Finished running tests with ios-sim"
else
    "${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
fi

3) Run your tests

xcodebuild \
    -sdk iphonesimulator6.1 \
    -project DC\ Wire\ Sizer.xcodeproj \
    -target DC\ Wire\ Sizer\ Tests \
    -configuration Debug \
    RUN_UNIT_TEST_WITH_IOS_SIM=YES 

...

Test Suite 'All tests' finished at 2013-06-06 16:03:35 +0000.
Executed 104 tests, with 0 failures (0 unexpected) in 0.991 (1.063) seconds

Finished running tests with ios-sim
Showing first 200 notices only


** BUILD SUCCEEDED **

Useful Commands:

xcodebuild -showsdks

$ xcodebuild -showsdks
OS X SDKs:
    Mac OS X 10.7                   -sdk macosx10.7
    OS X 10.8                       -sdk macosx10.8

iOS SDKs:
    iOS 6.1                         -sdk iphoneos6.1

iOS Simulator SDKs:
    Simulator - iOS 4.3             -sdk iphonesimulator4.3
    Simulator - iOS 5.0             -sdk iphonesimulator5.0
    Simulator - iOS 5.1             -sdk iphonesimulator5.1
    Simulator - iOS 6.0             -sdk iphonesimulator6.0
    Simulator - iOS 6.1             -sdk iphonesimulator6.1

xcodebuild -list -project

$ xcodebuild -list -project DC\ Wire\ Sizer.xcodeproj
Information about project "DC Wire Sizer":
    Targets:
        DC Wire Sizer
        DC Wire Sizer Tests
        In Work Unit Test

    Build Configurations:
        Debug
        Release_AppStore
        Release_TestFlight

    If no build configuration is specified and -scheme is not passed then "Release_AppStore" is used.

    Schemes:
        DC Wire Sizer
        DC Wire Sizer - App Store
        InWorkUnitTest

Application Test vs Logic Tests
The unit test target that is created for new projects is an Application Unit Test. It injects your test code into the app by setting BUNDLE_LOADER and TEST_HOST in the build settings. The test targets you create from the New Target menu are logic tests. Logic test are stand alone independent from your application.

This is my Bundle Loader and Test Host Values.

Bundle Loader: $(BUILT_PRODUCTS_DIR)/DC Wire Sizer.app/DC Wire Sizer
Test Host:     $(BUNDLE_LOADER)

Potential issue: if you have multiple versions of Xcode installed. Then you need to check your xcode-select setting:

gdunham: ~$ xcode-select -print-path
/Applications/Xcode.app/Contents/Developer