Chain automated builds in the same Docker Hub repository

Question

Due to build time restrictions on Docker Hub, I decided to split the Dockerfile of a time-consuming automated build into three files. Each one of those "sub-builds" finishes within Docker Hub's time limits.

I have now the following setup within the same repository:

| branch | dockerfile         | tag    |
| ------ | ------------------ | ------ |
| master | /step-1.Dockerfile | step-1 |
| master | /step-2.Dockerfile | step-2 |
| master | /step-3.Dockerfile | step-3 |

The images build on each other in the following order:

• step-1.Dockerfile : FROM ubuntu
• step-2.Dockerfile : FROM me/complex-image:step-1
• step-3.Dockerfile : FROM me/complex-image:step-2

A separate web application triggers the building of step-1 using the "build trigger" URL provided by Docker Hub (to which the {"docker_tag": "step-1"}' payload is added). However, Docker Hub doesn't provide a way to automatically trigger step-2 and then step-3 afterwards.

How can I automatically trigger the following build steps in their respective order?** (i.e., trigger step-2 after step-1 finishes. Then, trigger step-3 after step-2 finishes).

NB: I don't want to set up separate repositories for each of step-i then link them using Docker Hub's "Repository Links." I just want to link tags in the same repository.

Note: Until now, my solution is to attach a Docker Hub Webhook to a web application that I've made. When step-n finishes, (i.e., calls my web application's URL with a JSON file containing the tag name of step-n) the web application uses the "build trigger" to trigger step-n+1. It works as expected, however, I'm wondering whether there's a "better" way of doing things.

As requested by Ken Cochrane, here are the initial Dockerfile as well as the "build script" that it uses. I was just trying to dockerize Cling (a C++ interpreter). It needs to compile LLVM, Clang and Cling. As you might expect, depending on the machine, it needs a few hours to do so, and Docker Hub allows "only" 2-hour builds at most :) The "sub build" images that I added later (still in the develop branch) build a part of the whole thing each. I'm not sure that there is any further optimization to be made here.

Also, in order to test various ideas (and avoid waiting h-hours for the result) I have setup another repository with a similar structure (the only difference is that its Dockerfiles don't do as much work).

UPDATE 1: On Option 5: as expected, the curl from step-1.Dockerfile has been ignored:

Settings → Build Triggers → Last 10 Trigger Logs

| Date/Time                 | IP Address      | Status  | Status Description       | Request Body               | Build Request |
| ------------------------- | --------------- | ------- | ------------------------ | -------------------------- | ------------- |
| April 30th, 2016, 1:18 am | <my.ip.v4.addr> | ignored | Ignored, build throttle. | {u'docker_tag': u'step-2'} | null          |

Another problem with this approach is that it requires me to put the build trigger's (secret) token in the Dockerfile for everyone to see :) (hopefully, Docker Hub has an option to invalidate it and regenerate another one)

UPDATE 2: Here is my current attempt: It is basically a Heroku-hosted application that has an APScheduler periodic "trigger" that starts the initial build step, and a Flask webhook handler that "propagates" the build (i.e., it has the ordered list of build tags. Each time it is called by the webhook, it triggers the next build step).

Answer 1

I recently had the same requirement to chain dependent builds, and achieved it this way using Docker Cloud automated builds:

• Create a repository with build rules for each Dockerfile that needs to be built.

• Disable the Autobuild option for all build rules in dependent repositories.

• Add a shell script named hooks\post_push in each directory containing a Dockerfile that has dependents with the following code:

    for url in $(echo $BUILD_TRIGGERS | sed "s/,/ /g"); do
      curl -X POST -H "Content-Type: application/json" --data "{ \"build\": true, \"source_name\": \"$SOURCE_BRANCH\" }" $url
    done
  

• For each repository with dependents, add a Build Environment Variable named BUILD_TRIGGERS to the automated build, and set the Value to a comma-separated list of the build trigger URLs of each dependent automated build.

Using this setup, a push to the root source repository will trigger a build of the root image, once it completes and is pushed the post_push hook will be executed. In the hook a POST is made to each dependent repositories build trigger, containing the name of the branch or tag being built in the requests body. This will cause the appropriate build rule of the dependent repository to be triggered.

Answer 2

How long is the build taking? Can you post your Dockerfile?

Option 1: is to find out what is taking so long with your automated build to see why it isn't finishing in time. If you post it here, we can see if there is anything you can do to optimize.

Option 2: Is what you are already doing now, using a 3rd party app to trigger the builds in the given order.

Option 3: I'm not sure if this will work for you, since you are using the same repo, but normally you would use repo links for this feature, and then chain them, when one finishes, the next one triggers the first. But since you have one repo, it won't work.

Option 4: Break it up into multiple repos, then you can use repo links.

Option 5: Total hack, last resort (not sure if it will work). You add a CURL statement on the last line of your Dockerfile, to post to the build trigger link of the repo with the given tag for the next step. You might need to add a sleep in the next step to wait for the push to finish getting pushed to the hub, if you need one tag for the next.

Honestly, the best one is Option 1: what ever you are doing should be able to finish in the allotted time, you are probably doing some things we can optimize to make the whole thing faster. If you get it to come in under the time limit, then everything else isn't needed.

Answer 3

It's possible to do this by tweaking the Build Settings in the Docker Hub repositories.

First, create an Automated Build for /step-1.Dockerfile of your GitHub repository, with the tag step-1. This one doesn't require any special settings.

Next, create another Automated Build for /step-2.Dockerfile of your GitHub repository, with the tag step-2. In the Build Settings, uncheck When active, builds will happen automatically on pushes. Also add a Repository Link to me/step-1.

Do the same for step-3 (linking it to me/step-2).

Now, when you push to the GitHub repository, it will trigger step-1 to build; when that finishes, step-2 will build, and after that, step-3 will build.

Note that you need to wait for the previous stage to successfully build once before you can add a repository link to it.

Answer 4

I just tried the other answers and they are not working for me, so I invented another way of chaining builds by using a separate branch for each build rule, e.g.:

master              # This is for docker image tagged base
docker-build-stage1 # tag stage1
docker-build-latest # tag latest
docker-build-dev    # tag dev

in which stage1 is dependent on the base, the latest is dependent on stage1, and dev is based on the latest.

In each of the dependencies’ post_push hook, I called the script below with the direct dependents of itself:

#!/bin/bash -x

git clone https://github.com/NobodyXu/llvm-toolchain.git
cd llvm-toolchain
git checkout ${1}
git merge --ff-only master

# Set up push.default for push
git config --local push.default simple

# Set up username and passwd
# About the credential, see my other answer:
#     https://stackoverflow.com/a/57532225/8375400
git config --local credential.helper store
echo "https://${GITHUB_ROBOT_USER}:${GITHUB_ROBOT_ACCESS_TOKEN}@github.com" > ~/.git-credentials

exec git push origin HEAD

The variables GITHUB_ROBOT_USER and GITHUB_ROBOT_ACCESS_TOKEN are environment variables set in Docker Hub auto build configuration.

Personally, I prefer to register a new robot account with two-factor authentication enabled on GitHub specifically for this, invite the robot account to become a collaborator and use an access token instead of a password as it is safer than using your own account which has access to far more repositories than needed and is also easy to manage.

You need to disable the repository link, otherwise there will be a lot of unexpected build jobs in Docker Hub.

If you want to see a demo of this solution, check NobodyXu/llvm-toolchain.