6

I'm not so experienced but i worked on some big Java EE projects (using maven2) with very distinct ways to handle the installation / delivery on the different platforms.

1) One of them was to use snapshots for development and then make a maven release, of components and main webapplications. Thus the delivery is:

  • war/ear files
  • List item
  • properties files
  • sgdb files
  • some others

And teams will use that files to put the new application versions in the different platforms. I think this process is strict and permits you to always keep easily the different configurations passed in production, but it's not really flexible, the process is a bit heavy and it conducted us to sometimes do some dirty things like overriding a class of a war to patch a regression... This is an e-commerce website with 10million unique visitors per month and a 99.89% availability.

2) Another i saw is to checkout the sources on each platform and then install the snapshot artifacts in a local repository. Then the application server will use these snapshots of the .m2 folder. There is not a real delivery process since to put a new version in production, we just have to update the sources of the components / webapps, do some maven clean install and restart the application server. I think it's more flexible but i see some drawbacks and this approach seems dangerous for me. This website has a frontoffice, i don't know the numbers but it's far less than the 1st one. It also has a big backoffice available for most employees of a 130 000 people company.

I guess depending on the website, its exposition to the public and the availability required, we have to adapt the delivery strategy to the needs.

I'm not here to ask which solution is the best but wonder if you have seen different things, and which strategy you would use in which case?

Arjan Tijms
  • 37,782
  • 12
  • 108
  • 140
Sebastien Lorber
  • 89,644
  • 67
  • 288
  • 419

5 Answers5

3

Without dealing dealing web sites, I had to participate in release management process for various big (Java) projects in heterogeneous environment:

  • development on "PC", meaning in our case Windows -- sadly still Windows Xp for now -- (and unit testing)
  • continuous integration and system testing on linux (because they are cheaper to setup)
  • pre-production and production on Solaris (Sun Fire for instance)

The common method I saw was:

  • binary dependency (each project uses the binaries produced by the other project, not their sources)
  • no recompilation for integration testing (the jars produced on PC are directly used on linux farms)
  • full recompilation on pre-production (meaning the binary stored on the Maven repo), at least to make sure that everything is recompiled with the same JDK and the sale options.
  • no VCS (Version Control System, like SVN, Perforce, Git, Mercurial, ...) on a production system: everything is deployed from pre-prod through rsynch.

So the various parameters to take into account for a release management process is:

  • when you develop your project, do you depend directly on the sources or the binaries of the other projects?
  • where do you store your setting values?
    Do you parametrize them and, if yes, when do you replace the variables by their final values (only at startup, or also during runtime?)
  • do you recompile everything on the final (pre-production) system?
  • How do you access/copy/deploy on your production system?
  • How do you stop/restart/patch your applications?

(and this is not an exhaustive list.
Depending on the nature of the application release, other concerns will have to be addressed)

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Thanks for your point of view. Actually we also noticed that sometimes in the 1) case we were using a different JDK at compilation / runtime depending on the developer who compiled. But it wasn't a big matter since the major version of JDK was the same, i guess they don't break retrocompatibility on minor versions... :) – Sebastien Lorber Dec 30 '10 at 09:31
  • 1
    PC is NOT equivalent to "windows" (or x86/x64, either). – George Dec 31 '10 at 01:56
  • @George: true. I have edited my answer to clarify what "PC" stands for in my particular context. – VonC Dec 31 '10 at 10:59
  • @Sebastien: the bounty isn't over yet. Give it time and hopefully, others will contribute ;) – VonC Jan 04 '11 at 20:41
  • Note: for more on the dependency, see also http://stackoverflow.com/questions/871187/using-eclipse-with-large-workspaces/871235#871235 – VonC Jan 05 '11 at 09:45
  • the bountry expires soon :D up – Sebastien Lorber Jan 07 '11 at 13:29
3

The answer to this varies greatly depending on the exact requirements and team structures.

I've implemented processes for a few very large websites with similar availability requirements and there are some general principles I find have worked:

  • Externalise any config such that the same built artifact can run on all your environments. Then only build the artifacts once for each release - Rebuilding for different environments is time consuming and risky e.g. it not the same app that you tested
  • Centralise the place where the artifacts get built. - e.g. all wars for production must be packaged on the CI server (using the maven release plugin on hudson works well for us).
  • All changes for release must be traceable (version control, audit table etc.), to ensure stability and allow for quick rollbacks & diagnostics. This doesn't have to mean a heavyweight process - see the next point
  • Automate everything, building, testing, releasing, and rollbacks. If the process is dependable, automatable and quick the the same process can be used for everything from quick fixes to emergency changes. We use the same process for a quick 5 minute emergency fix and for a major release, because it is automated and quick.

Some additional pointers:

See my answer property-placeholder location from another property for a simple way to load different properties per environment with spring.

http://wiki.hudson-ci.org/display/HUDSON/M2+Release+Plugin If you use this plugin and ensure that only only the CI server has the correct credentials to perform maven releases, you can ensure that all releases are performed consistently.

http://decodify.blogspot.com/2010/10/how-to-build-one-click-deployment-job.html A simple way of deploying your releases. Although for large sites you will probably need something more complicated to ensure no downtime - e.g. deploying to half the cluster at a time and flip-flopping web traffic between the two halves - http://martinfowler.com/bliki/BlueGreenDeployment.html

http://continuousdelivery.com/ A good website and book with some very good patterns for releasing.

Hope this helps - good luck.

Community
  • 1
  • 1
Pablojim
  • 8,542
  • 8
  • 45
  • 69
  • thanks. Do you fix versions in the release dependencies or use maven ranges or something? Actually for my 1st exemple we had delivery projects that deploy on simple command line the artifact, env configuration, to the server we want. – Sebastien Lorber Jan 09 '11 at 22:29
  • We have dev, test, pre-prod and production environments. We have a rule that for pre-prod and prod all versions in dependencies must be fixed. – Pablojim Jan 10 '11 at 13:51
  • Anything else in particular you are looking for? – Pablojim Jan 21 '11 at 15:32
1

To add to my previous answer, what you are dealing with is basically a CM-RM issue:

  • CM (Change Management)
  • RM (Release Management)

In other words, after the first release (i.e. the main initial development is over), you have to keep making release, and that is what CM-RM is supposed to manage.

The implementation of the RM can be either 1) or 2) in your question, but my point would be to add to that mechanism:

  • proper CM in order to track any change request, and evaluate their impact before committing to any development
  • proper RM in order to be able to realize the "release" tests (system, performance, regression, deployment tests), and then to planify, schedule, perform and then monitor the release itself.
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
1

Without claiming it's a best solution, this is how my team currently does staging and deployment.

  • Developers initially develop at their local machine, the OS is free to choose, but we strongly encourage using the same JVM as will be used in production.
  • We have a DEV server where frequently snapshots of the code is being pushed to. This is simply a scp from the binary build produced from the IDE. We plan to build directly on the server though.
  • The DEV server is used for stakeholders to continuously peek along with development. By its very nature it's unstable. This is well known with all users of this server.
  • If the code is good enough, it's branched and pushed to a BETA server. Again, this is a scp of a binary build from the IDE.
  • Testing and general QA takes place on this BETA server.
  • Mean while, if any emergency changes should be necessary for the software currently in production, we have a third staging server called the UPDATE server.
  • The UPDATE server is initially only used to stage very small fixes. Here too we use scp to copy binaries.
  • After all testing is conducted on UPDATE, we copy the build from UPDATE to LIVE. Nothing ever goes to the live servers directly, it always goes via the update server.
  • When all testing is finalized on BETA, the tested build is copied from the beta server to the UPDATE server and a final round of sanity testing is performed. Since this is the exact build that was tested on the beta server, it is very unlikely that problems are found in this stage, but we uphold the rule that everything deployed to the live server should go via the update server and that everything on the update server should be tested before moving it on.

This sliding strategy allows us to develop for 3 versions in parallel. Version N that's currently in production and staged via the update server, version N+1 that will be the next major release that's about to be released and is staged on the beta server, and version N+2 that is the next-next major release for which development is currently underway and is staged on the dev server.

Some of the choices that we made:

  • A full application (an EAR) typically depends on artifacts from other projects. We choose to include the binaries of those other projects instead of building the whole thing from source. This simplifies building and gives greater assurance that a tested application is bundled with exactly the right versions of all its dependencies. The cost is that a fix in such a dependency has to be manually distributed to all applications that depend on it.
  • Configuration for every staging is embedded in the EAR. We currently use a naming convention and a script copies the right version of each configuration file to the right location. Parameterizing the path for each configuration file, e.g. by using a single {stage} placeholder in a root config file is currently being considered. The reason we store the config in the EAR, is because the developers are the ones who introduce and depend on configuration, so they should be the ones responsible for maintaining it (adding new entries, removing unused one, tweaking existing ones, etc).
  • We use a DevOps strategy for a deployment team. It consists of a person who is purely a developer, two persons who are both developer and operations and two persons who are purely operations.

Embedding the configuration in the EAR might be controversial, since traditionally operations needs to have control about e.g. the DB data sources being used in production (to what server it points to, how many connections a connection pool is allowed to have, etc). However, since we have persons on the development team who are also in operations, they are easily able to sanity check the changes made by other developers in the configuration while the code is still in development.

Parallel to the staging we have the continuous build server server doing a scripted (ANT) build after every check-in (with a maximum of once per 5 minutes), and runs unit tests and some other integrity tests.

It remains difficult to say whether this is a best-of-breed approach and we're constantly trying to improve our process.

Arjan Tijms
  • 37,782
  • 12
  • 108
  • 140
  • Thanks. I don't catch what you mean by "we choose to include the binaries". Actually when you use spring you use the binaries, when you use one of your component you get the binaries from an internal maven repository etc... who recompiles everything? Configuration in ear is often used but sometimes it's not so good since you can't change it at runtime without opening the ear (sometimes you just have to change a value at a given time). For db connection params you can also use the application server to handle that. – Sebastien Lorber Jan 09 '11 at 22:02
  • It's kinda hard to manage these properties... see http://stackoverflow.com/questions/4126154/tool-to-handle-properties-and-log4j-in-a-complex-environment – Sebastien Lorber Jan 09 '11 at 22:06
  • I meant the binaries of our own projects. We have for example an application called Foo and this makes use of a jar produced by some other project of us called Utils, which produces utils.jar. When we build Foo, we check it out from Hg and *this* will already include utils.jar. – Arjan Tijms Jan 09 '11 at 22:25
  • About the ear, we have chosen to deploy in exploded format. There is then very little difference between changing /server/default/deploy/db-ds.xml and /server/default/deploy/myear/db-ds.xml. – Arjan Tijms Jan 09 '11 at 22:27
  • Oh, and an important benefit of having the configuration for different stages in the EAR is that this typically encourages you to keep this file together with the code in the same VCS. You could of course also keep these files in a configuration management system backed by its own VCS, but this makes it harder for developers to add a new item. If development and operations are two totally separate teams and/or a single app is deployed by multiple clients, then such separation makes sense. But for us this is not the case. – Arjan Tijms Jan 09 '11 at 22:36
  • Actually it's common to use utils.jar, many people use maven in java and it always do that. Imagine the compilation time if you use many internal projects? while sometimes building a war could just be adding components binarie, few classes and a web deployment descriptor. It would also be dangerous to recompile the whole thing for me... with maven2 ranges you don't even have to change the utils.jar manually, just have to build the utils.jar, deploy it, and all components will (or not), use the new utils.jar – Sebastien Lorber Jan 10 '11 at 12:43
1

I am a big advocate of a single deployable containing everything (Code, Config, DB Delta, ...) for all environments, built and released centrally on the CI server.

The main idea behind this is that Code, Config & DB Delta are tightly coupled anyway. The code is dependent on certain properties being set in the config and some objects (tables, views, ...) being present in the DB. So why split this and spend your time tracking everything to make sure it fits together, when you can just ship it together in the first place.

Another big aspect is minimizing differences between environments, to reduce failure causes to the absolute minimum.

More details in my Continuous Delivery talk on Parleys: http://parleys.com/#id=2443&st=5

Axel Fontaine
  • 34,542
  • 16
  • 106
  • 137