Docker build fails for ARM images

Question

I try to build a docker image for multiple architectures on Travis-CI. This is working quite well for amd64 and i386 but fails for ARM.

The Dockerfile build on top of {ARCH}/nextcloud:apache which is build on top of php:7.3-apache-stretch which again uses debian:stretch-slim. So all the images use the same stack and should react similar.

.travis.yml

env:
  - TAG=i386     ARCH=i386
  - TAG=amd64    ARCH=amd64
  - TAG=armhf    ARCH=arm32v7
  - TAG=aarch64  ARCH=arm64v8

before_script:
  - docker run --rm --privileged multiarch/qemu-user-static:register --reset

script:
  - docker build --pull --build-arg ARCH=$ARCH -t escoand/nextcloud:$TAG nextcloud

Dockerfile

ARG ARCH

FROM ${ARCH}/nextcloud:apache

RUN apt-get update && apt-get install -y supervisor && \
    rm -rf /var/lib/apt/lists/* && \
    mkdir /var/log/supervisord /var/run/supervisord

As mentioned the build for i386 and amd64 works without problems. The ARM builds fail already with the first RUN command:

standard_init_linux.go:185: exec user process caused "no such file or directory"
The command '/bin/sh -c apt-get update && apt-get install -y supervisor &&     rm -rf /var/lib/apt/lists/* &&     mkdir /var/log/supervisord /var/run/supervisord' returned a non-zero code: 1

https://travis-ci.org/escoand/dockerfiles/jobs/562967055

For me this sounds like the /bin/sh is the problem but don't know how to handle this.

Answer 1

First of all, let's understand, what exactly you try to do by employing the following trick:

before_script:
  - docker run --rm --privileged multiarch/qemu-user-static:register --reset

When you ask the Linux kernel to run some executable file, it needs to know, how to load this specific file, and whether this file is compatible with current machine, or not. By default, the ELF binary compiled for, say, arm64v8 is rejected by the kernel, running on amd64 hardware.

However, the binfmt_misc feature of the kernel allows you to tell it, how to handle the executables it cannot usually handle on its own - this includes the cases when the kernel does not know the binary format or considers it incompatible with current machine.

What the container started from the image multiarch/qemu-user-static:register does? It registers new handlers for ELF binaries built for alternative architectures, for example:

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: 
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

When this handler is registered, kernel knows that if it faces the binary starting with the magic bytes, specified in the magic field (also taking into account the mask), it has to run /usr/bin/qemu-aarch64-static binary (the interpreter) and let it take care about the loading and running the requested binary.

The problem in your question is: where is the /usr/bin/qemu-aarch64-static interpreter, which knows how to run aarch64 binaries on amd64? There is no one, as the base image you use does not include it (I pulled and introspected the arm64v8/nextcloud:apache image manually to confirm this)!

As per manpage for execve(2), when the kernel can not load the interpreter, it returns the "ENOENT" (no such file or directory) error:

ERRORS
<...>
       ENOENT The file pathname or a script or ELF interpreter does not exist,
       or a shared library needed for the file or interpreter cannot be found.

So, this is what happens: you build the image and specify the base image, built for ARM machines. On the first RUN, the kernel tries to execute /bin/sh file from the image, finds the QEMU handler for this type of binaries, then looks for the interpreter, does not find it and fails, hence the "no such file or directory" error.

To resolve this, you have to use the base image which has QEMU installed (and, thus, has /usr/bin/qemu-aarch64-static inside), which would allow you executing RUNs.