How to manage stable binaries and avoid risk of CI rebuilds when install packaging?

Question

I am looking for a tool to manage the collection of binary files (input components) that make up a software release. This is a software product and we have released multiple versions each year for the last 20 years. The details and types of files may vary, but this is something many software teams need to manage.

What's a Software Release made of?

A mixture of files go into our software releases, including:

• Windows executables/binaries (40 DLLs and 30+ EXE files).
• Scripts used by the installer to create a database
• API assemblies for various platforms (.NET, ActiveX, and Java)
• Documentation files (HTML, PDF, CHM)
• Source code for example applications

The full collected files for a single version of the release are about 90MB. Most are built from source code, but some are 3rd party.

Manual Process

Long ago we managed this manually.

1. When starting each new release the files used to build the last release would be copied to a new folder on a shared drive.
2. The developers would manually add or update files in this folder (hoping nothing was lost or deleted accidentally).
3. The software installer script would be compiled using the files in this folder to produce a SETUP.EXE (output).
4. Iterate steps 2 and 3 during validation & testing until release.

Automatic Process

Some years ago we adopted CI (building our binaries nightly or on-demand).

We resorted to putting 3rd party binaries under version control since they usually don't change as often.

Then we automated the process of collecting & updating files for a release based on the CI build outputs. Finally we were able to automate the construction of our SETUP.EXE.

Remaining Gaps

Great so far, but this leaves us with two problems:

1. Rebuilding Assemblies The CI mostly builds projects when something has changed, but when forced it will re-compile a binary that doesn't have any code change. The output is a fresh build of a binary we've previously tested (hint: should we always trust these are equivalent?).
2. Latest vs Stable Mostly our CI machine builds the latest versions of each project. In some cases this is ok, but often we want to release an older tested or stable version. To do this we have separate CI projects for the latest and stable builds - this works but is clumsy.

Thanks for your patience if you've got this far :-)

I Still Haven't Found What I'm Looking For

After some time searching for solutions it seems it might be easier to build our own solution, but surely someone else has solved these problems before!?

What we want is a way to store and manage binary files (either outputs from CI, or 3rd party files) such that each is tagged with a version (v1.2.3.4) that allows:

• The CI to publish new versions of each binary (but reject rebuilt versions that already exist).
• The development team to make a recipe for a software release (kinda like NuGet packages.config) that specifies components to include:
  • package name
  • version
  • path/destination in the release folder
• The Automatic package script to use the recipe collect the required files, and compile the install package (e.g. SETUP.EXE).

I am aware of past debates about storing binaries in a VCS. For now I am looking for a better solution. That approach does not appear ideal for long-term ongoing use (e.g. how to prune old binaries)... amongst other issues.

I have tried some artifact repositories currently available. From my investigation these provide a solution for component/artifact storage and version control. However they do not provide tools for managing a list of components/artifacts to include in a software release.

Does anybody out there know of tools for this?

Have you found a way to get your CI infrastructure to address these remaining issues?

If you're using an artifact repository to solve this problem, how do you manage and automate the process?

Answer 1

This is a very broad topic, but it sounds like you want a release management tool (e.g. BuildMaster, developed by my company Inedo), possibly in conjunction with a package management server like ProGet (which you tagged, and is how I discovered this question).

To address some specific questions you have, I'll associate it with a feature that would solve the problem:

A mixture of files go into our software releases, including...

This is handled in BuildMaster with artifacts. This video gives a basic overview of how they are manually added to releases and deployed to a file system: https://inedo.com/support/tutorials/buildmaster/deployments/deploying-a-simple-web-app-to-iis

Of course, once that works to satisfaction, you can automate the import of artifacts from your existing CI tool, create them from a BuildMaster deployment plan itself, pull them from your package server, whatever. Down the line you can also have your CI tool call the BuildMaster release management API to create a release and automatically have it include all the artifacts and components you want (this is what most of our customers do now, i.e. have a build step in TeamCity create a release from a template).

Rebuilding Assemblies ... The output is a fresh build of a binary we've previously tested (hint: should we always trust these are equivalent?)

You can mostly assume they are equivalent functionally, but it's only the times that they are not that problems arise. This is especially true with package managers that do not lock dependencies to specific version numbers (i.e. NuGet, npm). You should be releasing exactly the same binary that was tested in previous environments.

[we want] the development team to make a recipe for a software release (kinda like NuGet packages.config) that specifies components to include:

This is handled with releases. A developer can choose its name, dates, etc., and associate it with a pipeline (i.e. a set of testing stages that the artifacts are deployed to), then can "click the deploy button" and have the automation do all the work.

Releases are grouped by "application", similar to a project in TeamCity. As a more advanced use case, you can use deployables. Deployables are essentially individual components of an application you include in a release; in your case the "Documentation" could be a deployable, and maybe contain an artifact of the .pdf and .docx files. Deployables from other applications (maybe a different team is responsible for them, or whatever) can then be referenced and "included" in a release, or you can reference ones from a past release.

Hopefully that provides some overview and fits your needs. Getting into this space is a bit overwhelming because there are so many terms, technologies, and methodologies, but my advice is to start simple and then slowly build upon it, e.g.:

• deploy a single, manually uploaded component through BuildMaster to a share drive, then manually deploy it from there
• add a deployment plan that imports the component
• add a second plan and associate it with the 2nd stage that takes the uploaded artifact and deploys it to the target, bypassing the need for the share drive
• add more deployment plans and associate them with pipeline stages and promote through them all to "close out" a release
• add an agent and deploy to that instead of the default localhost server
• add more components and segregate their deployment with deployables
• add event listeners to email team members at points in the process
• start adding approvals if you require gated "sign-offs"

and so on.