How do you continuously deploy a large system consisting of several applications

Question

At the moment we are deploying our whole application chain together and at once to production, because of the many dependencies that the systems have.

Our Scrum teams are business theme based in order to ensure real business value at the end of each Sprint with every user story, so it often happens, that user stories need changes in several applications.

And we have several Scrum teams, working on the same systems. Logically we end up acceptance testing everything in a huge acceptance and (semi automated) regression test.

But doing a big bang roll-out to production is very time consuming, error prone and not scalable anymore... (or is it?) With continuous deployment we would like to enable the team to self service a roll out to production, so business rolls out features when they want to, not based on an IT schedule.

But how do we manage to roll-out changes (code, DB scripts) that are distributed over several code bases and find a strategy in order to deal with the dependencies between applications?

What's the strategy to have scalable continuous deployment? And how do you transition to this point?

What do you think?

flurdy · Accepted Answer · 2012-02-10T16:36:58.377

2

(That is quite a few questions inside one big question.)

But I would refer to the Continuous Delivery book http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912/

Edit: (As commented you already read this book) Some suggestions which you may already do, but for others with similar issue:

Ensure less coupled interfaces by using duck typing.
Use feature toggles to gradual roll out of features across your inter-dependant applications.
Evolution scripts for the DBs
Version table in every DB
Symlinks is a neat way of toggling rollouts of standalone apps
Have a look at how other big enterprises rollout:
- Amazon use of new server per deployment:
- Netflix build demo by Carl Quinn using short lived cloud instances
- 50 times a day at IMVU and blog by Ries

But I have no solid solution to the inter-dependency auto-deploy strategy you actually ask for :|

edited Feb 10 '12 at 16:36

answered Feb 09 '12 at 17:05

flurdy

3,782
29
31

I know ;-) maybe i am asking too much. I read the book and its brilliant on how to deal with a single deployment pipeline, but i couldn't find too much about dealing with continuously deploying in several connected interdepend deployment pipelines. Or is it maybe impossible and we have to stick with iteratively deploying our current "single deployment" pipeline. I know its a lot to ask, but i am grateful for anyone pointing me into the right direction. – Emwee Feb 09 '12 at 17:25
You dont have a solid solution, because there probably is none, but you pointed me in the right direction woth some concrete strategy examples. Thanks! – Emwee Feb 10 '12 at 21:33

score 2 · Answer 2 · answered Feb 09 '12 at 17:54

There is no single silver bullet that will solve your problem but Kwatee (http://www.kwatee.net) can go a long way in the right direction. Kwatee deals with distributed/collaborative applications over multiple servers if needed and can take care of triggering pre-requisite DB ugrade scripts and the like using pre-deploy and post-deploy actions. You can also configure parameters for various deployment environments (dev, test, prod). Kwatee has a web interface which make configuration easy but you'd get the best of all worlds by including it (via python CLI commands or Ant task) in a continuous integration tool.

score 0 · Answer 3 · answered Feb 09 '12 at 16:46

0

I use CruiseControl for continuous integration. It's easy to set it up so that, whenever someone checks something into the trunk, a build and automatic regression test is triggered. If the build or test fails, all devs that committed code since the last build receive an email with the list of possible culprit revisions. Devs that break the build (though not the regression test) have to bring donuts the next day.

The specific tools you use for continuous integration and testing may depend on your language(s) and platforms, but the concept is the same. See http://cruisecontrol.sourceforge.net/

Hope it helps.

answered Feb 09 '12 at 16:46

Diego

18,035
5
62
66

thanks for your advice. I can see that i was not completely clear on our needs and i apologize for that. we use Jenkins for _continuous integration_ in a similar way and for one codebase its perfect. but we would like to go one step further in the direction of _continuous deployment_ of several codebases of several applications that are all in some way dependent. we got the toolstack kind of sorted out, but we are looking for a strategy how to transform our "all applications at once deployment" into several independent continuous deployments. – Emwee Feb 09 '12 at 17:20

score 0 · Answer 4 · edited May 23 '17 at 11:45

0

I answered a very similar question over here: Best-practice for continuous integration and deployment

That might be worth checking out.

edited May 23 '17 at 11:45

Community

1
1

answered Feb 10 '12 at 04:54

EricMinick

1,487
11
12

it was indeed well worth to check it out. Thanks! – Emwee Feb 10 '12 at 21:46

score 0 · Answer 5 · answered Feb 11 '12 at 16:29

I might misinterpret but I think what you are saying is acceptance testing for the whole system is hard and you want each Scrum team to be able to lift their own weight? I think even if each Scrum team can do some of the testing you cannot release without going through a system testing phase prior to release. In other words, system testing is a must but the frequency can be adjusted provided each component can be tested separately with substituted dependencies. If isolated testing and small scale testing can be done by separate Scrum team then system tests can be completed once every 2 to 3 sprint where the tester's focus is system testing while testers focus more on bug fixes.

How do you continuously deploy a large system consisting of several applications

5 Answers5