Golang cross compiling with CGO inside docker image

Question

Requirement: An application has to be containerised as a docker image and needs to support arm64 and amd64 architectures.

Codebase: It is a golang application that needs to make use of git2go library and must have CGO_ENABLED=1 to build the project. The minimum reproducible example can be found here on github.

Host machine: I am using arm64 M1 mac and docker desktop to build the app but the results are similar on our amd64 Jenkins CI build system.

Dockerfile:

FROM golang:1.17.6-alpine3.15 as builder

WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum

RUN apk add --no-cache libgit2 libgit2-dev git gcc g++ pkgconfig

RUN go mod download

COPY main.go main.go

ARG TARGETARCH TARGETOS

RUN CGO_ENABLED=1 GO111MODULE=on GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -tags static,system_libgit2 -a -o gitoperations main.go

FROM alpine:3.15 as runner

WORKDIR /
COPY --from=builder /workspace/gitoperations .
ENTRYPOINT ["/gitoperations"]

Build steps:

docker buildx create --name gitops --use
docker buildx build --platform=linux/amd64,linux/arm64 --pull .

This setup works but the build is taking way too long when building for different arch. The time difference between this specific build step: RUN CGO_ENABLED=1 GO111MODULE=on GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -tags static,system_libgit2 -a -o gitoperations main.go is always 10x longer when building for different arch:

example:

1. On arm64 M1 mac (without rossetta): Building arm64 executable takes ~30s and amd64 takes ~300seconds.
2. On our amd64 Jenkins CI system: Building arm64 executable takes 10x longer than building amd64 executable.

This build times can be seen by looking at the docker buildx build command output.
I believe (and I can most certainly be wrong) its happening because docker is using qemu emulation when building for a cpu architecture thats not the same as host machine's cpu arch. So I want to make use of golang cross-compilation capabilities to speed up the build times.

What I have tried: I thought of having a single builder stage in this dockerfile for arm and amd arch by trying this syntax:
FROM --platform=$BUILDPLATFORM golang:1.17.6-alpine3.15 as builder. But using the same docker build commands after making this change to dockerfile gives build errors, this is what I get when running on arm64 M1 mac:

 > [linux/arm64->amd64 builder 9/9] RUN CGO_ENABLED=1 GO111MODULE=on GOOS=linux GOARCH=amd64 go build -tags static,system_libgit2 -a -o gitoperations main.go:
#0 1.219 # runtime/cgo
#0 1.219 gcc: error: unrecognized command-line option '-m64'

After reading through golang CGO documentation I think this error is happening because go is not selecting the correct c compiler that is able to build for both architectures and I need to set the CC env variable to instruct go which c compiler to use.

Question: Am I right in assuming that qemu is causing the build time difference and it can be reduced by using golang's native cross-compilation functionality?
How can I make go build compile for amd64 and arm64 from any host machine using docker desktop as I dont have any experience working with C code and gcc and I am not sure what value I should set for CC flag in the go build command if I need to support linux/amd64 and linux/arm64?

Answer 1

To be able to compile C code on go you need to set CC variable to arm cross compiler. You can see your CC variable by go env. The error you have is related with the native compiler in the host system you use. You should apk add gcc-arm-none-eabi in your dockerfile. After you downloaded necessary cross compilation tools. You need to link your gcc command to compiler you have downloaded via command I mentioned. Then you should be able to compile your application for arm64.

Could you also share your go env output. You might have to edit GOGCCFLAGS variable too.

Answer 2

There actually is already a clean and neat solution in form of @tonistiigi/xx's xx . Dockerfile cross-compilation helpers. While I had came across this repository while looking into Docker's multi-arch building-related Github actions I initially totally missed its importance.

The documentation has a section dedicated to Go / Cgo that explains how to cross-compile using CGO on Alpine:

FROM --platform=$BUILDPLATFORM tonistiigi/xx AS xx

FROM --platform=$BUILDPLATFORM golang:alpine
RUN apk add clang lld
COPY --from=xx / /
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
ENV CGO_ENABLED=1
RUN xx-go build -o hello ./hello.go && \
    xx-verify hello

xx-apk installs target platform-specific packages, whereas apk installs the build platform-specific packages, as we're using FROM --platform=$BUILDPLATFORM ....

xx-go conveniently wraps the go command, supplying the required target-specific settings under the hood. And the final xx-verify is a nice touch to check that the resulting binary in fact is for the target platform, and not accidentally for the build platform if these differ.

For my @thediveo/lxkns service image, the build times for linux/amd64+arm64 on a free Github action runner have dropped from 25mins down to 7mins, so approx. 3.5× as fast.

Answer 3

On the arm64 machine or container, install x86_64-linux-gnu-gcc , debian:bookworm is what I'm using.

# apt-get install g++-x86-64-linux-gnu libc6-dev-amd64-cross
# export CC=x86_64-linux-gnu-gcc
# export CXX=x86_64-linux-gnu-g++

Maybe you need to install other amd64 libs on the arm64 platform

# dpkg --add-architecture amd64 
# apt-get update
# apt-get install libjpeg-dev:amd64