21

I have a bunch of Docker containers all hooked up using docker-compose (previously fig). I have found that services that advertise themselves as "Docker CI" services are usually just talking about using Docker to run their build workers.

I would like a continuous integration service that preserves the Docker build cache between runs, instead of spinning up a completely new worker/filesystem/build context. Otherwise, builds take absolutely forever. Ideally managed (unlike https://github.com/groupon/DotCi, which doesn't come with CircleCI's ready-to-go per-build database).

The only thing I've found is TeamCity (old, you can't version build tasks in a yaml file easily like you can with CircleCI, you have to maintain infrastructure, etc). Quay.io doesn't provision a database like CircleCI and Travis do, so you have to bifurcate dev and prod and install one inside the container and run unit tests during the container build (ew - you can't spawn into an unbuilt container to debug things!).

How do you run continuous integration on your Docker container cluster?

user3906849
  • 221
  • 2
  • 4
  • Note that it is now possible to build docker images on Travis as well, https://blog.travis-ci.com/2015-08-19-using-docker-on-travis-ci/ – cboettig Jul 19 '16 at 17:00

5 Answers5

15

CircleCI 1.0

CircleCI lets you build Docker containers as part of your build: https://circleci.com/integrations/docker

You are however right that this doesn't cache the Docker images - it's a bit tricky on our stack to do that right, but we are planning on addressing that soon.

In the meantime, a number of customers have cached it using CircleCI's built-in (non-Docker) caching, by using docker save/docker load: https://circleci.com/docs/docker#caching-docker-layers. It's not ideal, but it should tide you over until we're able to support this a lot better.

Updated for CircleCI 2.0

CircleCI 2.0 is Docker-centric. You can build Docker images, utilize layer caching, use Docker Compose, and even use official and custom Docker images as your build environment.

halfer
  • 19,824
  • 17
  • 99
  • 186
Paul Biggar
  • 27,579
  • 21
  • 99
  • 152
  • 8
    I wish `docker load` did that, but it doesn't preserve the build cache. `docker load` takes about as much time as pulling a fully build image from a private repository so it doesn't do the trick. – user3906849 Jan 29 '15 at 16:39
  • 2
    Paul would you have an update about the "planning on addressing that soon" part? It would be interesting. – user2707671 Feb 18 '16 at 14:16
  • Based on my testing, it's my understanding that even the `docker save/load` strategy as linked does not work (the docker build commands are fully run at each build), as per this comment here: https://discuss.circleci.com/t/docker-1-10-0-is-available-beta/2100/15 @paul-biggar, or anyone- is there a working example? – thisjustin Aug 19 '16 at 11:38
3

I'm currently using CircleCI, see http://tschottdorf.github.io/cockroach-docker-circleci-continuous-integration for a good guide to use the docker cache

shippable is also worth checking out, as their entire infrastructure is based around docker and orchestrating multiple docker containers with shippable formations.

d3ming
  • 8,496
  • 5
  • 31
  • 33
  • 1
    just a quick fix, looks like the URL for the blog post works only without the trailing slash as it will return 404: http://tschottdorf.github.io/cockroach-docker-circleci-continuous-integration – fagiani Oct 27 '16 at 14:40
2

Codeship now supports storing the Docker image cache and retreiving this in order to speed up your builds.

https://codeship.com/documentation/docker/caching/

Aidan Feldman
  • 5,205
  • 36
  • 46
0

There is a related question "How can I let the gitlab-ci-runner DinD image cache intermediate images?".

Even if our solution requires a minimal amount of infrastructure maintenace (we need to run our own runner), I think it is worth a look.

Basically we do not run Docker in Docker, but we are creating a separate VM for runners only. This has some implications about security and cleanup tasks, please see also the README of our runner.

Maybe this solution will become obsolete in the future as Docker support for various CI systems gets improved, but right now it's a good mixture between performance, maintenace and usability.

Community
  • 1
  • 1
schmunk
  • 4,708
  • 1
  • 27
  • 50
-2

Have you seen wercker? It supports "bring your own container", will cache your images (I think on a per-worker basis, but after a few builds usually the startup reduces to a few seconds with a custom image), and afaik also allows building Docker images.

Downside: it requires far too many GitHub permissions.

krlmlr
  • 25,056
  • 14
  • 120
  • 217