Docker build how to add libraries to golang build

Question

I am building a GO application dealing with Apache Pulsar. Go client requires C++ libraries, as asked by the Pulsar documentation (same thing for Kafka btw).

I want to package all this into a Container, the smallest possible. I usually use SCRATCH and copy the output from another golang based container. Unfortunately, I cannot get external libraries from this initial container:

FROM golang:latest as builder
ENV GO111MODULE=on \
CGO_ENABLED=0 \
GOOS=linux \
GOARCH=amd64                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
ARG DOCKER_GIT_CREDENTIALS
WORKDIR /builder
ADD . /builder
RUN git config --global credential.helper store && echo "${DOCKER_GIT_CREDENTIALS}" > ~/.git-credentials
RUN make build
RUN echo $(go list -m) && mv bin/$(go list -m) app

FROM SCRATCH
COPY --from=builder /builder/app app
EXPOSE 8080
ENTRYPOINT ["./app"]

Using this make the build fail, looking for missing symbols

/go/pkg/mod/github.com/apache/pulsar/pulsar-client-go@v0.0.0-20200118070759-21660e9402f8/pulsar/client.go:29:9: undefined: newClient
...

while local build works.

How to properly integrate the libraries I need?

Ming L. · Answer 1 · 2020-01-18T19:00:05.520

Because the library you use has C++ lib dependency, to properly build a Pulsar Golang client docker image, you need use Docker stage build. We have the exact use case. You need download and install Pulsar C++ lib for build and run-time image in Dockerfile.

This is our docker file https://github.com/kafkaesque-io/pulsar-beam/blob/master/Dockerfile . I also suggest to use Go Module to build and manage Go dependencies as part of Docker stage build. This is how we do it. I hope it helps.

RUN wget --user-agent=Mozilla -O apache-pulsar-client.deb "https://archive.apache.org/dist/pulsar/pulsar-2.4.1/DEB/apache-pulsar-client.deb"
RUN wget --user-agent=Mozilla -O apache-pulsar-client-dev.deb "https://archive.apache.org/dist/pulsar/pulsar-2.4.1/DEB/apache-pulsar-client-dev.deb"

RUN apt install -y ./apache-pulsar-client.deb
RUN apt install -y ./apache-pulsar-client-dev.deb

# Copy go mod and sum files
COPY go.mod go.sum ./

# Download all dependencies. Dependencies will be cached if the go.mod and go.sum files are not changed
RUN go mod download

I use the standard ubuntu image for build as well as run-time image. I understand you are looking for the smallest possible image size. ubuntu:18.04 has a smaller footprint. You can also try alpine that I have not tested yet.

By the way, Apache offers a separate native Pulsar Go client library with no C++ dependency. At the time of writing in January 2020, there are still missing features such as customized routing mode for partitioned topics. If these features are not required in your project, I suggest to try the native Go client library to avoid C++ dependency. We have plan to switch to the new native library soon for the same reason.

Thank you, this is exactly what I needed.For native client, I guess you talk about https://github.com/Comcast/pulsar-client-go. It is interesting too. I will give a try to all this. Thanks — fredczj, Jan 18 '20 at 20:09

Docker build how to add libraries to golang build

1 Answers1