Xcode5 Code Coverage (from cmd-line for CI builds)

Question

How can I generate code coverage with Xcode 5 and iOS7?

Prior to upgrading I was getting code coverage just fine. Now I can't see any *.gcda files being produced.

The cmd-line that I'm using is:

xcodebuild -workspace ${module.name}.xcworkspace test -scheme ${module.name} -destination OS=${module.sdk.version},name=iPad -configuration Debug

Works with AppCode

When I execute the tests via AppCode I can see *.gcda files being produced in ~/Library/Caches/appCode20/DerivedData. . . I need this to work for my Continuous Integration builds.

Works from Xcode IDE

Also works from Xcode IDE. . . is there a cmd-line that will produce coverage, or is this an Xcode bug?

this used to work fine with Xcode5-DP3 but got broken in more recent versions. i suspect that it's a bug, or might need some other additional (undocumented) flag to xcodebuild — Jan Berkel, Sep 11 '13 at 22:27
I've logged a radar ticket. Will update here when I get a response. — Jasper Blues, Sep 11 '13 at 23:43
Stucked with the same thing 2 days ago after installing Xcode 5 GM. Also I've updated app target to ios7, so there is no way to use trick with older simulators to generate reports. — velkopopovizky, Sep 16 '13 at 05:52
@velkopopovizky Do you mean the deployment target? You only need to set the deployment target down, you can still up the SDK. . (I assume you already meant this, but just making sure). — Jasper Blues, Sep 16 '13 at 06:35
Customer decided to support ios7 and above, so we temporary switched deployment target to ios7, but I think I'll kill myself in this case. — velkopopovizky, Sep 16 '13 at 09:54
@JasperBlues I'm using ocunit2junit. And I've changed the invocation method from: _xcodebuild -workspace WORKSPACE_NAME.xcworkspace -scheme LogicTests -sdk iphonesimulator7.0 build | ./ocunit2junit_ to: **xcodebuild -workspace WORKSPACE_NAME.xcworkspace -scheme LogicTests -sdk iphonesimulator7.0 test | ./ocunit2junit** so, problem was - xcode 5 can't generate test reports via "build" build action. need to specify build action as "test" — velkopopovizky, Sep 16 '13 at 09:58
@velkopopovizky I *am* specifying the build action as test. In my case they're "application" tests. . . so it works for you with logic tests? — Jasper Blues, Sep 16 '13 at 10:54
@JasperBlues yes, I have separate target and scheme "LogicTests" with key TestAfterBuild=YES. BTW, "LogicTests" is not launching the app itself. — velkopopovizky, Sep 16 '13 at 11:55
It actually doesn't work for me from Xcode - I can generate coverage for my test files but not for the files I test, which are in a static library. — Sulthan, Sep 24 '13 at 08:38
@Sulthan - I believe my Typhoon project (Github) is set up to test via a static library, and has coverage. . Using the workaround below, I think. — Jasper Blues, Sep 24 '13 at 08:56
@JasperBlues Well, it worked for me on iOS 6 but I would like to have a solution not a workaround. — Sulthan, Sep 24 '13 at 08:59
With Xcode5 and running application tests I can't generate the .gcda files. What do I need to set in project settings or any where else to generate them ? — ArdenDev, Sep 27 '13 at 23:09
@IphoneDeveloper - To enable coverage in Xcode5 its the same as Xcode4. In a targets 'Build Settings' set 'Generate Test Coverage Files=YES', 'Instrument Program Flow=YES'. Once enabled you can proceed to the step that Sulthan outlined, so that .gcda files are properly flushed during testing. — Jasper Blues, Sep 28 '13 at 01:14

score 15 · Answer 1 · edited Sep 28 '13 at 03:55

15

The following is a fix for SenTestKit - simply add this class to your Tests target. Something similar should be possible to do with XCTest

@interface VATestObserver : SenTestLog

@end

static id mainSuite = nil;

@implementation VATestObserver

+ (void)initialize {
    [[NSUserDefaults standardUserDefaults] setValue:@"VATestObserver" forKey:SenTestObserverClassKey];

    [super initialize];
}

+ (void)testSuiteDidStart:(NSNotification*)notification {
    [super testSuiteDidStart:notification];

    SenTestSuiteRun* suite = notification.object;

    if (mainSuite == nil) {
        mainSuite = suite;
    }
}

+ (void)testSuiteDidStop:(NSNotification*)notification {
    [super testSuiteDidStop:notification];

    SenTestSuiteRun* suite = notification.object;

    if (mainSuite == suite) {
        UIApplication* application = [UIApplication sharedApplication];
        [application.delegate applicationWillTerminate:application];
    }
}

and add

extern void __gcov_flush(void);

- (void)applicationWillTerminate:(UIApplication*)application {
    __gcov_flush();
}

Why is this working?

Tests and the tested application are compiled separately. Tests are actually injected into the running application, so the __gcov_flush() must be called inside the application not inside the tests.

The little magic with the observer only enables us to check when the tests are going to end and we trigger __gcov_flush() to be called inside the app.

edited Sep 28 '13 at 03:55

Jasper Blues

28,258
22
102
185

answered Sep 24 '13 at 09:17

Sulthan

128,090
22
218
270

I'm trying to verify this as an accepted answer, however it seems the cmd-line for tests has stopped working in the GM release! Tried: xcodebuild -workspace ${module.name}.xcworkspace test -scheme ${module.name} -destination OS=7.0,name=iPad -destination-timeout=10 -configuration Debug . . will accept as soon as I can figure out how to run my test target through my build script. – Jasper Blues Sep 24 '13 at 10:08
It turns out that this also **fixes** AppCode. . when running via iOS7 simulator in AppCode, the usually lovely Test Reporter shows no results (only evidence of running tests is log statements). . however, adding this fix, also fixes the test report. . . unfortunately, however the test name appears twice. . still, nice work! – Jasper Blues Sep 24 '13 at 10:32
@JasperBlues I modified my answer a bit, maybe it will be better for you now. – Sulthan Sep 24 '13 at 12:30
3

Just tried calling __gcov_flush() from +testSuiteDidStop: without -applicationWillTerminate:. It worked for unit tests. – ZhangChn Sep 27 '13 at 10:02
5

is __gcov_flush defined in some library or header ? I get link errors for this – ArdenDev Sep 29 '13 at 03:54
That happens if you are compiling without code coverage enabled. – Sulthan Sep 29 '13 at 09:38
I've tried implementing this solution for XCTest, but get duplicate symbol error. I've added my own question to the site as this is already solved. http://stackoverflow.com/questions/19136767/generate-gcda-files-with-xcode5-ios7-simulator-and-xctest – MdaG Oct 02 '13 at 12:55
@Sulthan - Would you like to add an answer also here? Same solution I think but deals with gcovr : http://stackoverflow.com/questions/19112782/code-coverage-with-gcovr-and-xcode-5-is-not-working/19730601#19730601 – Jasper Blues Nov 01 '13 at 16:08
@JasperBlues I am not using gcovr. I have my own bash script to convert gcda to cobertura. Gcovr probably returns zero because there are no files generated, that's all. – Sulthan Nov 01 '13 at 20:31
@Sulthan, not using gcovr either. But from what I can see the it also converts .gcda files to a report, so on iOS7 there'd be the same problem of it not flushing, hence no data files. I figured someone should answer that question or close it as a duplicate. – Jasper Blues Nov 02 '13 at 02:36

Jasper Blues · Answer 2 · 2013-08-23T06:14:45.467

(This is not the answer, but a work-around . . .I'm still very much interested in a better solution)

Use iOS 6.1 Simulator

If you're targeting iOS 6.1 or earlier as a deployment target, you can use the 6.1 simulator.

Install the iOS6.1 Simulator via preferences/downloads
Use the following cmd-line:

xcodebuild -workspace ${module.name}.xcworkspace test -scheme ${module.name} -destination OS=6.1,name=iPad -configuration Debug

score 1 · Answer 3 · answered Oct 10 '13 at 15:33

We found that we had to add a bit of code to get the gcda files to flush from the system.

Code addition is to add extern void __gcov_flush(); to the top of your file and then call __gcov_flush(); just before the entire test suite exits.

Full explanation is here: http://www.bubblefoundry.com/blog/2013/09/generating-ios-code-coverage-reports/

score 1 · Answer 4 · edited Dec 18 '13 at 11:39

With the information from here I was able to craft this version which is the least invasive I could think of. Just add to your unit tests and run the tests as normal. The ZZZ ensures it is the last run suite of tests.

I had to ensure I added the GCC_GENERATE_TEST_COVERAGE_FILES and GCC_GENERATE_TEST_COVERAGE_FILES compiler flags to my test unit target too to get the coverage out.

//
//  Created by Michael May
//

#import <SenTestingKit/SenTestingKit.h>

@interface ZZZCodeCoverageFixForUnitTests : SenTestCase

@end

@implementation ZZZCodeCoverageFixForUnitTests

// This must run last

extern void __gcov_flush();

-(void)testThatIsntReallyATest
{
    NSLog(@"FLUSHING GCOV FILES");

    __gcov_flush();
}


@end

Edit, or another approach by Jasper:

I stripped the VATestObserver from the other answer down to this:

@interface VATestObserver : SenTestLog
@end

@implementation VATestObserver

extern void __gcov_flush(void);

- (void)applicationWillTerminate:(UIApplication*)application
{
   __gcov_flush();
  [super applicationWillTerminate:application];
}

@end

I stripped the VATestObserver from @Sulthan's excellent answer down too. . . otherwise it was causing some issues with the AppCode test reporter. . . Hope you don't mind, I've edited you answer to include that approach too. — Jasper Blues, Dec 18 '13 at 11:45

score 0 · Answer 5 · answered Dec 18 '13 at 05:03

Some more documentation here:

https://code.google.com/p/coverstory/wiki/UsingCoverstory

and some source code to use:

https://code.google.com/p/google-toolbox-for-mac/source/browse/#svn%2Ftrunk%2FUnitTesting

You need GTMCodeCoverageApp.h/.m and GTMCodeCoverageTestsXC.h/.m or GTMCodeCoverageTestsST.h/.m depending on if you are using XCTest or SenTest.

score 0 · Accepted Answer · edited May 23 '17 at 11:59

0

Update: New accepted answer

In some cases the coverage flushing needs to be done from within the app itself. The solution's outline in this question provide details.

edited May 23 '17 at 11:59

Community

1
1

answered Jan 11 '14 at 06:19

Jasper Blues

28,258
22
102
185

Xcode5 Code Coverage (from cmd-line for CI builds)

6 Answers6

Linked