Using docker-compose in a GitLab CI pipeline

Question

I'm trying to implement a GitLab continuous integration (CI) pipeline with the following .gitlab-ci.yml file:

image: docker:latest

# When using dind, it's wise to use the overlayfs driver for
# improved performance.
variables:
  DOCKER_DRIVER: overlay

services:
  - docker:dind

before_script:
  - docker info
  - curl -L "https://github.com/docker/compose/releases/download/1.10.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  - chmod +x /usr/local/bin/docker-compose
  - export PATH=/usr/local/bin:$PATH
  - echo $PATH

stages:
  - build
  - test

build:
  stage: build
  script:
    - docker-compose build
    - docker-compose up -d

test:
  stage: test
  script:
    - cd tests/tester
    - pytest test_run_tester.py
  except:
    - /^.*skip_tests$/

However, in GitLab I'm getting the following error message:

Running with gitlab-ci-multi-runner 1.10.4 (b32125f)
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:latest ...
Running on runner-2e54fd37-project-13-concurrent-0 via scw-de9c9c...
Fetching changes...
HEAD is now at 2504a08 Update CI config
From https://lab.iperlane.com/gio/ipercron-compose
   2504a08..5c2f23f  CI         -> origin/CI
Checking out 5c2f23f1 as CI...
Skipping Git submodules setup
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.8.14-docker-2
Operating System: Alpine Linux v3.5 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.751 GiB
Name: 90395a030c02
ID: 7TKR:5PQN:XLFM:EJST:NF2V:NLQC:I2IZ:6OZG:TR4U:ZEAK:EVXE:HIF7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
$ curl -L "https://github.com/docker/compose/releases/download/1.10.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   600    0   600    0     0   1175      0 --:--:-- --:--:-- --:--:--  1176

  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  3 7929k    3  254k    0     0   116k      0  0:01:08  0:00:02  0:01:06  258k
 17 7929k   17 1358k    0     0   433k      0  0:00:18  0:00:03  0:00:15  704k
 61 7929k   61 4861k    0     0  1164k      0  0:00:06  0:00:04  0:00:02 1636k
100 7929k  100 7929k    0     0  1703k      0  0:00:04  0:00:04 --:--:-- 2300k
$ chmod +x /usr/local/bin/docker-compose
$ export PATH=/usr/local/bin:$PATH
$ docker-compose build
/bin/sh: eval: line 51: docker-compose: not found
ERROR: Build failed: exit code 127

The system seems not to be able to find docker-compose, even after installing it and adding it to the path. Is there perhaps another image such that docker-compose commands can be made, or a different way to install it in the .gitlab-ci.yml?

Have you tried setting an absolute path to `docker-compose`? — Fairy, Feb 17 '17 at 15:10
@Kurt Peek: What of the three Setup types did you use: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#runner-configuration — Max Schindler, May 08 '17 at 09:24
This question is related to https://stackoverflow.com/questions/39868369/run-docker-compose-build-in-gitlab-ci-yml/41455005#41455005 — tmaier, Oct 04 '17 at 07:44

score 22 · Accepted Answer · answered Feb 19 '17 at 02:07

The problem

This is complex problem.

The docker:latest image is based on alpine (Alpine Linux), which is built using musl-libc. This system is very barebones, and as such doesn't have everything a full-fledged desktop Linux might have. In fact, dynamic executables need to be compiled specifically for this system.

docker-compose is a Python application, bundled into a Linux executable using PyInstaller. These executables are really only expected to be able to run on the system which they were built. If you run an alpine container, and apk add file, you can see that the (dynamically-linked) executable you downloaded is expecting a specific interpreter.

# file /usr/local/bin/docker-compose
/usr/local/bin/docker-compose.pyinstaller: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=00747fe5bcde089bb4c2f1ba7ab5348bc02ac5bf, stripped

The problem is /lib64/ld-linux-x86-64.so.2 doesn't exist in alpine. In fact, /lib64 doesn't even exist.

The solution

Because docker-compose is a Python application, it can also be installed using pip:

Testing this inside an alpine container:

$ docker run --rm -it alpine /bin/bash

Here are the commands you can run manually, or add to your .gitlab-ci.yml before_script:

# apk add --no-cache python py2-pip
...
# pip install --no-cache-dir docker-compose
...
# docker-compose -v
docker-compose version 1.11.1, build 7c5d5e4

It turns out there is actually an open issue for this:

https://github.com/docker/compose/issues/3465

Great explanation. I created a simple docker image to shave off precious time from the install process. See https://hub.docker.com/r/tmaier/docker-compose/ — tmaier, Oct 04 '17 at 07:45

score 18 · Answer 2 · edited May 08 '21 at 18:24

18

Docker-compose is now an official image. You can put

image: 
  name: docker/compose:1.23.2
  entrypoint: [""]

at the top of your .gitlab-ci.yml. You need to remove the default entrypoint of the compose image, which is done by the second line. After that, you can use both docker and docker-compose.

This works per GitLab 9.4. You can also build a custom image

FROM docker/compose:1.23.2
ENTRYPOINT []

and use it as your image.

edited May 08 '21 at 18:24

Lord Elrond

13,430
7
40
80

answered Nov 13 '19 at 14:53

Marco de Wild

189
1
3

This doesn't appear to start the docker daemon, resulting in `docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))`. – Eric H. May 26 '21 at 00:22
That is correct. Using a docker image (or derived images like compose) will only supply the docker command line utility. You still need to set up the docker daemon, which is usually done by having `docker:dind` (docker-in-docker) as a service. Your job then connects to the daemon running in another container (but within the same network). I was playing around with our setup, but I could not reproduce your exact error. – Marco de Wild May 27 '21 at 05:03
2022 Update: it's abandoned, but as per other answer, the main `docker` image now has `docker-compose` (newer versions) – declension Dec 05 '22 at 10:40

score 2 · Answer 3 · answered Nov 18 '22 at 16:35

The docker Docker image now has docker-compose preinstalled.

It can be pulled to your local machine simply with docker pull docker.

It can be used in GitLab CI like this:

my_job:
  stage: my_stage
  image: docker
  script:
    - docker-compose up

It is based on Alpine Linux, so apk commands are the order of the day. It comes with /bin/sh preinstalled.

score 0 · Answer 4 · answered Jun 15 '22 at 18:02

I was fighting with this few days. Finally I found and tested solution: https://hub.docker.com/r/tmaier/docker-compose/ with open source code: https://github.com/tmaier/docker-compose.

Below is sample from source above:

image: tmaier/docker-compose:latest

services:
  - docker:dind

before_script:
  - docker info
  - docker-compose --version

build image:
  stage: build
  script:
    - docker-compose build

and everything is working (no need to set up extra Docker-in-Docker). The only disadvantage is that it is not verified publisher in hub.docker.com.

Using docker-compose in a GitLab CI pipeline

4 Answers4

The problem

The solution

Linked