GitLab CI/CD: building multiarch Docker images

Question

I want an easy way to build multiarch Docker images in a GitLab runner. By easy, I mean that I just would have to add a .gitlab-ci.yml in my project and it would work.

Here is the .gitlab-ci.yml that I wrote. It builds a multiarch image using buildx and then pushes it to the GitLab registry:

image: cl00e9ment/buildx

services:
- name: docker:dind

variables:
  PLATFORMS: linux/amd64,linux/arm64
  TAG: latest

before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

build:
  stage: build
  script:
  - docker buildx build --platform "$PLATFORMS" -t "${CI_REGISTRY_IMAGE}:${TAG}" . --push

The problem is that the linux/arm64 platform isn't available.

Here is how I built the cl00e9ment/buildx image (strongly inspired from snadn/docker-buildx):

Here is the Dockerfile:

FROM docker:latest

ENV DOCKER_CLI_EXPERIMENTAL=enabled
ENV DOCKER_HOST=tcp://docker:2375/

RUN mkdir -p ~/.docker/cli-plugins \
  && wget -qO- https://api.github.com/repos/docker/buildx/releases/latest | grep "browser_download_url.*linux-amd64" | cut -d : -f 2,3 | tr -d '"' | xargs wget -O ~/.docker/cli-plugins/docker-buildx \
  && chmod a+x ~/.docker/cli-plugins/docker-buildx
RUN docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
RUN docker context create buildx \
    && docker buildx create buildx --name mybuilder \
    && docker buildx use mybuilder
RUN docker buildx inspect --bootstrap

...add here is the .gitlab-ci.yml file used to build and push the cl00e9ment/buildx image:

image: docker:latest

services:
  - name: docker:dind

before_script:
  - docker login -u cl00e9ment -p "$DOCKER_HUB_TOKEN"

build:
  stage: build
  script:
  - docker build --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host -t cl00e9ment/buildx .
  - docker run --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host cl00e9ment/buildx docker buildx inspect --bootstrap
  - docker push cl00e9ment/buildx

test:
  stage: test
  script:
  - docker run --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host cl00e9ment/buildx docker buildx inspect --bootstrap

So what's happening?

• At the end of the build, in the Dockerfile, I run docker buildx inspect --bootstrap to list the available platforms. It gives linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6. So it's all good.
• After that, I run it again (just after the build and just before the push) and it still gives linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.
• However, on the test stage, when the image is freshly downloaded from Docker Hub on a clean environment, it gives linux/amd64, linux/386.

Why?

Answer 1

There is a lot of outdated and incorrect information on building multiarch images on GitLab CI unfortunately. The seems to change quite frequently as it's still an experimental feature. But as of the time of this post, this is how I got my multiarch build working on GitLab public runners (armv6, armv6, arm64, amd64):

First, one must build and push a Docker image containing the buildx binary. Here is the Dockerfile I am using for that:

FROM docker:latest
ARG BUILDX_VER=0.4.2
RUN mkdir -p /root/.docker/cli-plugins && \
    wget -qO ~/.docker/cli-plugins/docker-buildx \
    https://github.com/docker/buildx/releases/download/v${BUILDX_VER}/buildx-v${BUILDX_VER}.linux-amd64 && \
    chmod +x /root/.docker/cli-plugins/docker-buildx

The current GitLab runner image does not initialize the binfmt handlers correctly despite running the initialization code: https://gitlab.com/gitlab-org/gitlab-runner/-/blob/523854c8/.gitlab/ci/_common.gitlab-ci.yml#L91

So we have to do it in our pipeline. We refer to the comments in MR 1861 of the GitLab Runner code and add in the following magic sauce to our .gitlab-ci.yml:

before_script:
  - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes 

Then we can run the rest of our pipeline script with docker login, docker buildx build --use, docker buildx build --push ... and so on.

Now the runner is ready to build for multiple architectures. My final .gitlab-ci.yml can be seen here: https://github.com/oofnikj/nuttssh/blob/multiarch/.gitlab-ci.yml

Answer 2

Ok, I think I know whats going on here: you need to call update-binfmts --enable somewhere to enable the extra formats provided by binfmt_misc for .

I was able to get multiarch images working with buildx on gitlab-ci (after lots of searching) using this repo and its docker images: https://gitlab.com/ericvh/docker-buildx-qemu

However that repo has self dependency on its own docker image repository to build multiarch versions of itself AND it depends on a gitlab-ci template repo for its ci. I'm not super confident in how this web of dependency all began and the owner of that repo is far more skilled than me, but for my uses, I've forked the repo and I'm now trying to modify its CI to be less dependent on external sources.

EDIT: For people from the future this is the Dockerfile:

FROM debian
# Install Docker and qemu
# TODO Use docker stable once it properly supports buildx
RUN apt-get update && apt-get install -y \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg2 \
        software-properties-common && \
    curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - && \
    add-apt-repository "deb https://download.docker.com/linux/debian $(lsb_release -cs) stable" && \
    apt-get update && apt-get install -y \
        docker-ce-cli \
        binfmt-support \
        qemu-user-static

# Install buildx plugin
RUN mkdir -p ~/.docker/cli-plugins && \
    ARCH=`dpkg --print-architecture` && echo Running on $ARCH && curl -s https://api.github.com/repos/docker/buildx/releases/latest | \
        grep "browser_download_url.*linux-$ARCH" | cut -d : -f 2,3 | tr -d \" | \
    xargs curl -L -o ~/.docker/cli-plugins/docker-buildx && \
    chmod a+x ~/.docker/cli-plugins/docker-buildx

# Write version file
RUN printf "$(docker --version | perl -pe 's/^.*\s(\d+\.\d+\.\d+.*),.*$/$1/')_$(docker buildx version | perl -pe 's/^.*v?(\d+\.\d+\.\d+).*$/$1/')" > /version && \
    cat /version

And a stripped down version of .gitlab-ci.yml

build:
  image: docker:dind
  stage: build
  services:
    - name: docker:dind
      entrypoint: ["env", "-u", "DOCKER_HOST"]
      command: ["dockerd-entrypoint.sh"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    # See https://github.com/docker-library/docker/pull/166
    DOCKER_TLS_CERTDIR: ""
  before_script:
    - |
      if [[ -z "$CI_COMMIT_TAG" ]]; then
        export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
        export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
      else
        export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE}
        export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_TAG}
      fi
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
  script:
    - docker build -t "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG" -t "$CI_APPLICATION_REPOSITORY:latest" .
    - docker push "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"  
    - docker push "$CI_APPLICATION_REPOSITORY:latest"

EDIT:

Further, I've found that this gitlabci configuration that uses the image built above can use the build cache:

stages:
  - build

variables:
  CI_BUILD_ARCHS: "linux/amd64,linux/arm/v6,linux/arm/v7"
  CI_BUILD_IMAGE: "registry.gitlab.com/gdunstone/docker-buildx-qemu"

build_master:
  image: $CI_BUILD_IMAGE
  stage: build
  services:
    - name: docker:dind
      entrypoint: ["env", "-u", "DOCKER_HOST"]
      command: ["dockerd-entrypoint.sh"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    # See https://github.com/docker-library/docker/pull/166
    DOCKER_TLS_CERTDIR: ""
  retry: 2
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  # Use docker-container driver to allow useful features (push/multi-platform)
    - update-binfmts --enable # Important: Ensures execution of other binary formats is enabled in the kernel
    - docker buildx create --driver docker-container --use
    - docker buildx inspect --bootstrap
  script:
    - >
        docker buildx build --platform $CI_BUILD_ARCHS 
        --cache-from=type=registry,ref=$CI_REGISTRY_IMAGE/cache:latest
        --cache-to=type=registry,ref=$CI_REGISTRY_IMAGE/cache:latest
        --progress plain 
        --pull --push 
        --build-arg CI_PROJECT_PATH=$CI_PROJECT_PATH
        -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" 
        -t "$CI_REGISTRY_IMAGE:latest" .
  only:
    - master