How to reduce multistage build duplicate steps time cost issue?

Question

I have a go application, which depends on cgo. When build, it needs libsodium-dev, libzmq3-dev, libczmq-dev, and when run it also needs above three packages.

Currently, I use next multistage build: a golang build environment as the first stage & a debian slim as the second stage. But you could see the 3 packages installed for two times which waste time(Later I may have more such kinds of package added).

FROM golang:1.12.9-buster AS builder

WORKDIR /src/pigeon

COPY . .

RUN apt-get update && \
  apt-get install -y --no-install-recommends libsodium-dev && \
  apt-get install -y --no-install-recommends libzmq3-dev && \
  apt-get install -y --no-install-recommends libczmq-dev && \
  go build cmd/main/pgd.go


FROM debian:buster-slim

RUN apt-get update && \
  apt-get install -y --no-install-recommends libsodium-dev && \
  apt-get install -y --no-install-recommends libzmq3-dev && \
  apt-get install -y --no-install-recommends libczmq-dev && \
  apt-get install -y --no-install-recommends python3 && \
  apt-get install -y --no-install-recommends python3-pip && \
  pip3 install jinja2

WORKDIR /root/

RUN mkdir logger

COPY --from=builder /src/pigeon/pgd .
COPY --from=builder /src/pigeon/logger logger

CMD ["./pgd"]

Of course, I can give up multi-stage build, just use golang1.12.9-buster for build, and continue for run, but this will make final run image bigger (which is the advantage of multi-stage build).

Do I miss something or I had to make choice between above?

Just see some guys suggest [this](https://stackoverflow.com/questions/53281570/how-to-copy-library-files-between-stages-of-a-multi-stage-docker-build-while-pre) to me, but one package install may results in other dependency package install, really hard for me to copy all packages from one stage to another stage although I could use `dpkg -L` to find all things. — atline, Sep 05 '19 at 09:37

score 2 · Accepted Answer · edited Sep 06 '19 at 05:50

this is my take about your question:

FROM debian:buster-slim as base

RUN mkdir /debs /debs_tmp \
    && chmod 777 /debs /debs_tmp

WORKDIR /debs
RUN apt-get update \
    && apt-get install -y -d \
            --no-install-recommends \
            -o dir::cache::archives="/debs_tmp/" \
        libsodium-dev \
        libzmq3-dev \
        libczmq-dev \
    && mv /debs_tmp/*.deb /debs \
    && rm -rf /debs_tmp \
    && apt-get install -y --no-install-recommends \
        python3 \
        python3-pip \
    && pip3 install jinja2 \
    && rm -rf /var/lib/apt/lists/*


##################

FROM golang:1.12.9-buster AS builder

COPY --from=base /debs /debs
WORKDIR /debs

RUN dpkg -i *.deb

WORKDIR /src/pigeon
COPY . .

RUN go build cmd/main/pgd.go

##################

FROM base

RUN rm -rf /debs

WORKDIR /root/
RUN mkdir logger

COPY --from=builder /src/pigeon/pgd .
COPY --from=builder /src/pigeon/logger logger

CMD ["./pgd"]

You can download the required packages in a temporary folder, move the debs in a new location and finally COPY the debs in the next stage. Finally you simply use the first image you've created.

BTW the containers will run as root. This might be an issue depending on what the software does, you might want to consider to use a user without "powers".

EDIT: sorry for the edits but I ran a couple of example locally and didn't have a go script ready.

Similar to buildkit apt cache, but this solution no need `experimental` support. — atline, Sep 06 '19 at 05:53

BMitch · Answer 2 · 2019-09-05T09:57:34.550

At the COPY . . step, any time your source changes, the cache will bust and you will run all later steps again. You can reorder the steps to allow docker to cache the install of your dependencies. You can also join the apt-get install commands into one to reduce overhead of processing the package manager db.

FROM golang:1.12.9-buster AS builder

WORKDIR /src/pigeon

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
     libsodium-dev \
     libzmq3-dev \
     libczmq-dev

COPY . .

RUN go build cmd/main/pgd.go


FROM debian:buster-slim

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
     libsodium-dev \
     libzmq3-dev \
     libczmq-dev \
     python3 \
     python3-pip \
 && pip3 install jinja2

WORKDIR /root/

RUN mkdir logger

COPY --from=builder /src/pigeon/pgd .
COPY --from=builder /src/pigeon/logger logger

CMD ["./pgd"]

You will still install the packages twice, but now those installs are cached for future builds. The way to reuse the install of the libraries is to reorder the steps, installing the libraries in a common base image, and then install the go compiler on your build stage, but that will almost certainly be more overhead than installing libraries twice.

With BuildKit, you could share the apt cache between builds using an experimental syntax, but this requires that all builds use BuildKit (the syntax is not backwards compatible), and modifying docker's Debian image to preserve the apt package cache. From the BuildKit experimental documentation, there's the following example for apt:

# syntax = docker/dockerfile:experimental
FROM ubuntu
RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt \
  apt update && apt install -y gcc

https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md

How to reduce multistage build duplicate steps time cost issue?

2 Answers2