0

Since I've installed go-sqlite3 as dependency in my go project my docker build time started oscillating around 1 min.

I tried to optimize the build by using go mod download to cache dependencies But it didn't reduce overall build time.

Then I found out that

go-sqlite3 is a CGO enabled package you are required to set the environment variable CGO_ENABLED=1 and have a gcc compile present within your path.

So I run go install github.com/mattn/go-sqlite3 as an extra step and it reduced build time to 17s~

I also tried vendoring, but it didn't help with reducing the build time, installing library explicitly was always necessary to achieve that.

## Build
FROM golang:1.16-buster AS build
WORKDIR /app
 
# Download dependencies
COPY go.mod .
COPY go.sum .
RUN go mod download

RUN go install github.com/mattn/go-sqlite3 //this reduced build time to around 17s~

COPY . .

RUN go build -o /myapp

But somehow I am still not happy with this solution. I don't get why adding this package makes my build so long and why I need to explicitly install it in order to avoid such long build times. Also, wouldn't it be better to install all packages after downloading them?

Do you see any obvious way of improving my current docker build?

michal.materowski
  • 437
  • 3
  • 13
  • have you considered multi stage docker ? otherwise, https://stackoverflow.com/questions/24601619/how-to-improve-golang-compilation-speed / https://stackoverflow.com/questions/24341654/go-build-became-very-slow-after-installing-a-new-version-of-go / https://stackoverflow.com/questions/32061750/go-build-rebuilds-unnecessarily/47109826 –  Jul 21 '21 at 10:54
  • I do use multi-stage docker, although that's not relevant for speed of go build itself. When I read suggested answers it seems, that installing package explicitly resolves package's dependencies. This step is already present in my question, did I miss something or what did you mean with your multi stage docker build? – michal.materowski Aug 03 '21 at 12:45
  • the question seeks to trace the execution of his setup to understand why it is slow and how to imprve it. –  Oct 03 '21 at 11:07
  • @michal.materowski: I suspect that "multi-stage Docker" comes with the implicit recommendation that layer caching is available in your build environment - so SQLite is built in an earlier stage, and rebuilds very quickly because the original layers are available. – halfer Oct 03 '21 at 16:27
  • Indeed, I would go one step further and build your own parent image, which rebuilds weekly on a schedule, and pushes that image to your own private registry. Then in your app project, you can pull a custom image containing SQLite and you don't even need multi-stage Docker to improve build times. – halfer Oct 03 '21 at 16:29
  • 1
    Thanks for Your input @halfer, seems like a reasonable improvement. Although I only work locally on my own pc (it's a private after-hours project). It does not seem like it would make much difference when comparing private registry base image to cached layer approach, although I'll keep your tip in mind for my future battles with Docker – michal.materowski Oct 05 '21 at 16:36

1 Answers1

1

The fact of the matter is that the C-based SQLite package just takes a long time to build. I use it myself currently, and yes it's painful every time. I have also been unhappy with it, and have been looking for alternatives. I have been busy with other projects, but I did find this package QL [1], which you can build without C [2]:

go build -tags purego

or if you just need read only, you can try SQLittle [3].

  1. https://pkg.go.dev/modernc.org/ql
  2. https://pkg.go.dev/modernc.org/ql#hdr-Building_non_CGO_QL
  3. https://github.com/alicebob/sqlittle
Zoe
  • 27,060
  • 21
  • 118
  • 148
Zombo
  • 1
  • 62
  • 391
  • 407
  • That's pretty helpful to know, that there are alternatives. I've tried out QL, but I'm not happy with it (syntax and tooling) and I do need to write to DB, so option [3] is also ruled out. Thanks for Your input though and I'm sorry to hear that go-sqlite3 is also annoying – michal.materowski Aug 03 '21 at 12:53