29

Is there a way/plugin to integrate the new Xcode service and/or the new Apple CI with Jenkins?

Why?

A main issue with having a Jenkins server + an OSX build slave connected via ssh is that Unit Tests do not work, as the iOS Simulator needs a graphical environment which is not present in this configuration.

I hope that it is possible to integrate the Xcode service (which supports Unit Testing) with Jenkins.

It could be that using the Apple CI will be enough for my needs, but this question aims at the integration of the Xcode service with Jenkins.

What I do already know

I have experience with the existing Xcode Jenkins plugin, but it seems not to support the brand new Xcode service or the new Apple CI. I'm especially keen on unit testing via CI (which did not work properly over a ssh session with the old way).

What I want to know

I'd like info on the following issues currently not working with Jenkins and an ssh connected build slave:

  • Unit Tests on a headless system
  • Acceptance tests with Frank or similar
  • Automatic Provisioning Profile updating (Apple CI does that)

And info on things that currently do work fine with Jenkins and an ssh connected build slave and still should work with an Xcode service integration:

  • Builds of different build configurations (Release, Debug, TestFlight) / schemes
  • Automatic Build number increment
    • (With Jenkins I can set the build number in my project to ${BUILD_NUMBER}, and Jenkins sets this environment variable according to its build number. When the Apple CI does the builds most probably it will set the build number instead.)
    • Handling/synchronization of Build number between Jenkins & Apple CI
  • Accessing build products of the Apple CI from different Jenkins Jobs
    • e.g. for a Job to upload to TestFlight
  • Backup of builds
  • Automatic builds on git push to a specific branch
  • E-Mail notifications

Some additional questions/hints

  • I'm not sure whether the Apple CI == Xcode service or if the Apple CI just uses the Xcode service. In the latter case the Xcode service just would be like an intelligent build slave, and Jenkins maybe could use that to do builds and tests, but manage build numbers and products by itself.
  • I'm aware that the Apple CI is an separate CI, and integrating several CIs with each other is not the most easy or useful way to go. I just fear that the Apple CI is not flexible enough for my needs (see above), and that the old way with Jenkins bears some problems (see above).
fabb
  • 11,660
  • 13
  • 67
  • 111
  • You are not going to find very many people who can answer this on the iOS tag because we don't get OSX Mavericks unless we are members of the Mac developer program. I am adding the "osx" tag. – Andrew Aug 05 '13 at 13:57
  • https://devforums.apple.com/thread/198677 – fabb Aug 10 '13 at 14:42
  • If you still want to integrate jenkins, maybe mention that in your post. – Andrew Aug 10 '13 at 15:19
  • Also maybe take a look [here](https://devforums.apple.com/message/822234#822234). You might want to move your post to a different topic (xcode beta or mavericks beta) – Andrew Aug 10 '13 at 15:28
  • Thanks for the link! What do you mean by "move to a different topic"? Change the title or tags? – fabb Aug 11 '13 at 06:51
  • Well currently your post on dev forums is under mac development > server. Other posts about this are under xcode beta or mavericks beta. – Andrew Aug 11 '13 at 13:07
  • Ah ok, I thought you mean this stackoverflow post... – fabb Aug 11 '13 at 14:36
  • Moved it there: https://devforums.apple.com/thread/198801 – fabb Aug 11 '13 at 14:39

2 Answers2

14

I believe you are going to have to choose either jenkins or xcode server, not both. I don't know much about xcode server, but I do know about jenkins and xcode 5.

Builds with different configurations:

In the xcode plugin, you can set the scheme to use.

Automatic Build Number Increment

I added a parameter to my jenkins job called XCODEBUILDNUMBER. And whenever I start a build, I simply copy the build number out of my xcode project (I increment it manually. Mine looks like 080813A) and paste it into the XCODEBUILD parameter. I use this to name my output files, etc. There are plugins for jenkins that can automatically increment your build number, but they don't integrate, or sync with xcode.

Handling/synchronization of Build number between Jenkins & Apple CI

As I said before, I don't know of a way to sync the build numbers, but I just thought of a possible solution. You could use the command line tool plistbuddy, to set the build number in your info.plist, as a build step in your jenkins job.

Unit Tests

I have not successfully made unit tests work with Xcode5, but I know that the xcode plugin for jenkins supports it. I believe that the absence of the "Test After Build" key in the project settings may have something to do with it. If you make it work, i'd love to know. (I am also keen on making this work)

Acceptance Tests

From what I can tell, Frank is a command line tool. You can easily integrate it into your Jenkins job, and I believe that it will fail the build if your tests don't pass.

Accessing build products of the Apple CI from different Jenkins Jobs

Not completely sure what you mean, but with jenkins you can archive your build product (a .ipa), for later download and upload to a service like testflight. Again, I don't know much about Xcode Server (CI).

Backup of builds

As I said before, jenkins can archive your build product. Also, I use the the ${BUILD_NUMBER} variable in my build products directory, so I have a different directory for each build. This directory is also backed up to my Time Machine, and important builds copied to my web directory.

Automatic builds on git push to a specific branch

With the jenkins git plugin, you can make jenkins poll your scm in a interval specified by you, and can trigger a build on a change.

E-Mail notifications

I am sure that there is a plugin for this. (that emails you when a build failed/succeeded. in fact, this may be built-in)

In Closing

The xcode CI is a full independent CI, that may be hard to integrate with jenkins. Personally, I would recommend jenkins simply due to its extendability. Sorry I don't know much about Xcode Server.

Andrew
  • 15,357
  • 6
  • 66
  • 101
  • Thanks for your extensive answer, but I already know how to use Jenkins with a regular OSX build slave with the xcode-plugin. This has several drawbacks (unit tests do not work, as they need a simulator which needs a logged in user, which doesn't work over ssh; same problem with frank; no automatic provisioning profile updates etc). So I think Jenkins can access the Xcode *service* to work around some of these problems. Maybe I mixed up the Xcode service with the Apple CI, I'll clarify my question. – fabb Aug 08 '13 at 15:15
  • In order for anything to interface with the xcode service (sorry, i might have been using the wrong term before), there will probably need to be a command line interface for the xcode service. Is there one? – Andrew Aug 08 '13 at 15:20
  • I don't have enough information on this as I do not yet have access to the OSX Mavericks beta :-( – fabb Aug 08 '13 at 15:26
  • I don't either. From what I can tell, it does not have a command line interface, but a slick web interface. (im jealous) – Andrew Aug 08 '13 at 15:30
  • I think the xcode service will do everything you want (including unit testing). Why do you need jenkins to be integrated with it? – Andrew Aug 08 '13 at 15:38
  • I do not need to use Jenkins, but I doubt that the Apple CI will be flexible enough to do everything I listed in my question. I might be wrong, the documentation yet is pretty sparse. – fabb Aug 08 '13 at 15:50
  • I still think that you will probably have to pick one or the other, and that integrating the xcode service into jenkins is probably not possible. What do you think the xcode service can't do? – Andrew Aug 08 '13 at 15:53
  • And yes, the docs are pretty sparse from what I can tell, but it should be pretty self-explanatory. – Andrew Aug 08 '13 at 15:54
  • E.g. the TestFlight uploads, or Frank integration tests. – fabb Aug 09 '13 at 06:02
  • If there are command line script that need to be run (could you write one for frank or test flight uploads?), it can run them as a part of the build, test, or archive steps in your build scheme. Even without xcode service. – Andrew Aug 09 '13 at 13:59
  • A big issue with the current way is that the **simulator cannot be run** when I've got a Jenkins server and a separate OSX build slave **connected via ssh**. That's a main reason I'm looking for ways to integrate Jenkins with the new Xcode service. – fabb Aug 10 '13 at 07:45
  • That is probably your biggest issue. If you use the xcode service by itself, my understanding is that it will be on a computer with xcode on it (so it could run unit tests). Also, if you got a good look at the xcode service (or apple ci), and could tell if you could trigger builds with scripts or another means, then you could potentially trigger xcode service builds with Jenkins (which would make xcode service do unit tests, and frank tests if you integrated that into your build scheme), archive the build artifacts (with Jenkins), and do whatever else you want jenkins to do. – Andrew Aug 10 '13 at 12:47
  • Yepp, that would be awesome. If possible. – fabb Aug 10 '13 at 13:02
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/35198/discussion-between-fabb-and-santa-claus) – fabb Aug 10 '13 at 13:45
  • Thanks for your effort, you've earned the bounty. I'm not yet accepting the answer though, as I'm still waiting for some first-hand experience. – fabb Aug 11 '13 at 18:08
  • You will need both Jenkins if you have some sort of source code repo that is hosted on another server like TFS. Then you'd need Jenkins to launch scripts to get/commit them into Xcode's Git. Also this is for synchronizing builds if you want. – mskw Nov 08 '13 at 23:08
6

I've got unit tests running in Jenkins with Xcode 5 on my OS X build slave. Instead of using the Xcode plugin, I run as an execute shell build step:

xcodebuild test -scheme <scheme> -configuration Coverage -sdk iphonesimulator7.0 -destination OS=7.0,name="iPhone Retina (4-inch)"

My coverage configuration is the exact same as my Debug config, except Generate Test Coverage Reports is set to YES, and Instrument Program Flow is set to YES. This is done so test coverage files are created. Due to a bug in Xcode 5, I call __gcov_flush(); in the tearDown of all my tests. I pipe the output of this xcodebuild command into ocunit2junit to get test reports in Jenkins.

Mark
  • 7,167
  • 4
  • 44
  • 68
  • 1
    do you use an `ssh` connection between the jenkins server and the osx build slave? – fabb Dec 13 '13 at 11:47
  • I'll have to try this out sometime. Looks promising, especially since the xcode plugin has repeatedly failed to impress me. – Andrew Dec 14 '13 at 16:27
  • Yes, but I need to pipe the output to ocunit2junit, so I execute it in a separate build step. – Mark Dec 17 '13 at 19:35
  • can you explain "pipe the output of this xcodebuild command into ocunit2junit to get test reports in Jenkins." I always get this error `Build step 'Execute shell' marked build as failure` although all tests are Passed – sahara108 May 27 '14 at 10:52
  • do I need Apple hardware to run the OS X build slave? – Onnmir Aug 25 '14 at 15:03
  • @sgh2105 I believe so. – Mark Aug 25 '14 at 17:08