7

I am currently automating our iOS testing on jenkins. So far I had no problems with running unit tests, converting OCUnit into JUnit and generating code coverage in Cobertura format (lots of googling but no hard problems).

However, when creating UIAutomation job, I am stuck. The tests are running successfully (calling instruments from command line). Generating junit report was tricky but possible. The problem is that when running UIAutomation, no code coverage files are generated.

Is there a possibility to generate them? If not, could you please explain why?

What I tried so far:

  1. Attaching UIAutomation to an already running application on iOS simulator.
    • this seems impossible. Either the running application is killed by instruments and a new instance is relaunched or a cryptic error message is printed.
  2. Quit simulator at the end of js script using a combination of osascript and UIAHost. performTaskWithPathArgumentsTimeout.
    • application ended gracefully but no coverage generated.

Edit After some testing with a different framework I realized it's not enough to quit the simulator, you have to explicitely call exit() from the application. With UI Automation this is a bit tricky but you can declare an applicaton scheme, e.g. my-app://exit and call it through MobileSafari using UIAHost.performTaskWithPathArgumentsTimeout. Will check whether this is enough for the files to be generated.

Sulthan
  • 128,090
  • 22
  • 218
  • 270
  • Hi, I know this might not the answer: You can also try http://testingwithfrank.com I tested the UI of my application to 80%. It is also easy to read because it use cucumber – Vinh Nov 21 '12 at 15:08
  • @VinhTran Well, there are many different frameworks I could use, including OCUnit, but we have chosen UIAutomation and we already have a lot of tests written. – Sulthan Nov 22 '12 at 14:24
  • @VinhTran In the end, I have started rewriting the tests to Frank. It goes slowly because I am not a ruby guy but the biggest pro is that the framework is open. I have already fixed some bugs and I can easily check why a test doesn't work. Jenkins integration is much easier and CC generation works without problems. – Sulthan Apr 25 '13 at 09:46
  • I'm happy to hear that it could help you. The really disadvantage is only visible at the beginning. But later on the benefits will be clearly visible. – Vinh Apr 26 '13 at 10:18
  • @VinhTran The test organization is better immediately (the possibility to tag scenarios/features is very nice when debugging, e.g. "@record" tag to record videos). Also the possibility to call obj-c methods directly helps me test things I couldn't test with UI Automation. The biggest problem until now are the ocassional bugs in Frank - however, it's open-source, some of my fixes were already merged :) – Sulthan Apr 26 '13 at 10:35

2 Answers2

2

Following these steps, I was able to generate the code coverage files from UI Automation and display the information through the cobertura Jenkins plugin.

First set the “Generate Test Coverage Files” and “Instrument Program Flow” build settings to Yes. This will generate code coverage files every time you run your application in the simulator and exit the application. Add UIApplicationExitsOnSuspend in your Info.plist file and set this option to 'YES'. Run the UI automation test and at the end of it you can exit the app either by manually pressing the HOME button in the simulator or using the UIATarget.localTarget().deactivateAppForDuration() method. Note if your app has any UI Automation tests that rely on the deactivateAppForDuration() method, the tests will terminate upon running the command.

Once you have the gcda files you can generate the cobertura xml file by downloading gcovr (https://software.sandia.gov/trac/fast/wiki/gcovr) and running the command

gcovr -r your_root_directory --object-directory path_to_gcda_files --xml > coverage.xml

With that you can setup the Jenkins cobertura plugin to display the information as needed.

Source: http://blog.octo.com/en/jenkins-quality-dashboard-ios-development/#step2-2

Ed-E G
  • 331
  • 5
  • 7
  • Yes, that confirms that the suggestion at the end of my question is indeed valid. – Sulthan Jul 09 '13 at 15:29
  • @Sulthan didnt get what you meant by this comment. also can you tell if the above link works - UIAutomation + Code coverage ? – Tatvamasi Nov 20 '13 at 08:48
0

From my understanding code coverage files get generated when the app quits, but you can't just kill the simulator.

Have you tried creating a separate target for your app where you have the info.plist property "UIApplicationExitsOnSuspend" set to true?

There is a slightly wider problem, however. The generated coverage files aren't cumulative and get overwritten each time the application quits. So, depending on how your tests are structured (i.e. are you killing and starting the app for each distinct test) then you may struggle to get decent code coverage.

JonB
  • 4,422
  • 2
  • 27
  • 25
  • I am not `killing` the simulator. I tried to use an `osascript` which quits the simulator - that's very different. `UIApplicationExitsOnSuspend` is not a good solution because the `suspend` event (sending app to background) is a valid use case and shouldn't end the app. However, I did try it. Also, the overwrite of code coverage files is expected and is not a problem. – Sulthan Dec 31 '12 at 15:36
  • I get that but code coverage only gets written on the app exiting, so sounds like you're stuck (until apple improve this). – JonB Jan 02 '13 at 11:21
  • I don't think the problem is caused by app exiting. I am able to make the app exit normally. I believe the problem is caused by the way the app is __launched__, not exited. Unfortunately, I can't control how instruments launch it. – Sulthan Jan 02 '13 at 11:33