6

I need your recommendations for continuous build products for a large (1-2MLOC) software development project. Characteristics:

  • ClearCase revision control
  • Approx 80% C++; 15% Java; 5% script or low-level
  • Compiles for Green Hills Integrity OS, but also some windows and JVM chunks
  • Mostly an embedded system; also includes some UI pieces and some development support (simulation tools, config tools, etc...)
  • Each notional "version" of the deliverable includes deployment images for a number of boards, UI machines, etc... (~10 separate images; 5 distinct operating systems)
  • Need to maintain/track many simultaneous versions which, notably, are built for a variety of different board support packages
  • Build cycle time is a major issue on the project, need support for whatever features help address this (mostly need to manage a large farm of build machines, I guess..)
  • Operates in a secure environment (this is a gov't program) (Edited to add: This is a classified program; outsourcing the build infrastructure is a non-starter.)

Interested in any best practices or peripheral guidance you might offer. The build automation issues is one of several overlapping best practices that appear to be missing on the program, but try to keep your answers focused on build infrastructure piece and observations directly related.

Cost is not the driving concern. Scalability and ease of retrofitting onto an existing infrastructure are key.

(Edited to address @Dan's comment. ;-)

andersoj
  • 22,406
  • 7
  • 62
  • 73
  • 1
    Sigh. This is a **government program** and **cost is not an object** – Dan Dec 22 '10 at 01:39
  • 2
    @Dan: There is plenty to be exasperated about here. But please understand that significant $ spent on getting the basics right (i.e., continuous build infrastructure) will save us $ in the long run. In the best of all worlds, I represent the good guy, trying to improve on a decidedly bad situation. If you haven't had the opportunity, find a government program and pick up a shovel... – andersoj Dec 22 '10 at 05:37

3 Answers3

3

From my experience with similar systems, there are approximately two parts to this problem:

  • A repeatable method for checking out sources, building the software, and testing it (if you want to do continual testing as well as building), using a small number of command-line invocations.

  • A means of calling these command lines on various servers in the build farm.

For the latter, we've been using BuildBot, which seems to work pretty well.

For the former, we have a homegrown solution that started out as a simple bash shell script and grew ... rather substantially. From experience, I'd suggest starting out in python rather than bash -- you'll spend far more code in handling setup and configuration than in actually invoking programs. (Also, it's probably easier to run it on Windows if you're doing that.)

The things I've found to be really key in our script's usefulness are:

  • Ironclad repeatability. We have a standard set of build tools, and the scripts start out by scrubbing environment variables. There are very few command-line options; everything goes into configuration files, and those go in version control.

  • Logging. We produce a log of every command that the build script executes.

  • Configuration file inheritance. Each variant of our software gets a configuration file, and those files can include more-general settings (which include even-more-general settings).

  • Extensibility. When we add a new source component, it's pretty easy to add a set of instructions for building that component (and the instructions can be arbitrary bash code). The "can be arbitrary code" part is probably key here; no way is a pre-existing product going to be able to do all of the quirky things that you need for a large complex real-world system.

You can get started with a reasonably simple script and let it grow organically as the need arises; honestly, although ours is a bit messy, I think we got a much more usable result that way than we would have with heavy top-down design.

Brooks Moses
  • 9,267
  • 2
  • 33
  • 57
2

Cost isn't an object? I've worked for GreenHills, and they've solved these issues for their in-house build/test farms. Ask them to do the same for you.

Phil Miller
  • 36,389
  • 13
  • 67
  • 90
  • Are you talking about literal outsourcing of the build to Green Hills owned build farms (not possible here) or that Green Hills has customized build products that could be hosted in our development facility? – andersoj Dec 21 '10 at 19:42
  • Well, cost is always an object, but in this case the need is urgent enough that money will be found to make this happen. I'm standing in the way trying to make sure the best decision is made. (And obviously I don't know as much as I should...) – andersoj Dec 21 '10 at 20:44
  • I'm suggesting asking their field engineers to set up your build farm for you. – Phil Miller Dec 24 '10 at 05:56
2

When I see emphasis on things like scalability and security in a build system, I start thinking that you might be a candidate for the enterprise class build systems / CI systems. Conveniently, it sounds like you can afford them as well. A year old SD Times article provides a basic breakdown between the enterprise and team level build tools.

My company makes AnthillPro and we've worked with a number of companies on large embedded projects as well as highly secure projects. IBM is probably the largest other player in the space with BuildForge.

AnthillPro puts some extra emphasis on what you do with the images in the minutes/hours/days post build (do you install them onto simulators / hardware and run automated tests? stage them? promote them?) but we also see folks using it for just build.

EricMinick
  • 1,487
  • 11
  • 12
  • Excellent, I was aware of Anthill before, but hadn't tracked progress. Any comments about the specific issues of building with a focus on embedded, integrity OS, multiple target platforms, and ClearCase integration? We're definitely checking on the IBM product as well, but I'm looking for the right value proposition for the program and it's not yet clear. – andersoj Dec 22 '10 at 17:19
  • Clearcase is definitely not a problem. UCM, Base, snapshot, dynamic, weird settings, we've seen a lot. In general, we're pretty agnostic to build technology and have a bunch of customers working on embedded stuff. Multiple platforms / Itegrity OS shouldn't be a problem either. The agents in our build grid are (almost) pure Java and work just about anywhere. Integrity looks like it has decent Java support. If you want to go into more detail, feel free to hit me at eric@urbancode.com. – EricMinick Dec 27 '10 at 18:25