Running multiple projects using docker which each runs with docker-compose

Question

We are using microservices approach to build our product. We are using some projects which each uses docker-compose to run. The problem is that in development environment, if we want to change codes in multiple projects and test developed codes, we must run projects separately and link them together manually.

Now we want to create a development kit which clones projects and runs them together and handles links. Can docker-compose handle multiple docker-compose file? If not is there any sufficient tool to do that for us? Or is there any recommended approach for our goal?

EDIT: For example we have two projects: PROJECT_A and PROJECT_B. Each one has its own docker-compose.yml and each one needs postgresql to run. We have docker-compose.yml in PROJECT_A like this:

db:
    image: postgres:9.4
    ports:
      - "5432"

project_a:
    build: .
    command: python2.7 main.py
    links:
        - db

And we have docker-compose.yml in PROJECT_B like this:

db:
    image: postgres:9.4
    ports:
      - "5432"

project_b:
    build: .
    command: python2.7 main.py
    links:
        - db

Each project can run separately and works fine. But if we want to change the api between PROJECT_A and PROJECT_B we need to run both projects and link them together to test our code. Now we want to write a development kit project which can run both projects and link them if needed. What is the best approach to do this?

score 10 · Answer 1 · edited Jul 06 '19 at 20:23

You can do this by combining services from multiple files using the extends feature of docker-compose. Put your projects in some well-defined location, and refer to them using relative paths:

../
├── foo/
│   └── docker-compose.yml
└── bar/
    └── docker-compose.yml

foo/docker-compose.yml:

base:
    build: .

foo:
    extends:
        service: base
    links:
        - db

db:
    image: postgres:9

If you wanted to test this project by itself, you would do something like:

sudo docker-compose up -d foo

Creating foo_foo_1

bar/docker-compose.yml:

foo:
    extends:
        file: ../foo/docker-compose.yml
        service: base
    links:
        - db

bar:
    build: .
    extends:
        service: base
    links:
        - db
        - foo

db:
    image: postgres:9

Now you can test both services together with:

sudo docker-compose up -d bar

Creating bar_foo_1
Creating bar_bar_1

Please note that keyword extends is no longer supported from docker-compose v3 — Lewis, Sep 26 '18 at 09:54

score 3 · Answer 2 · answered Jan 31 '16 at 13:19

Am not 100% sure on your question so this will be a wide answer.

1) Everything can be in the same compose file if it's running on the same machine or server cluster.

#proxy
haproxy:
  image: haproxy:latest
  ports:
    - 80:80


#setup 1
ubuntu_1:
  image: ubuntu
  links:
    - db_1:mysql
  ports:
    - 80

db1:
  image: ubuntu
  environment:
    MYSQL_ROOT_PASSWORD: 123


#setup 2
ubuntu_2:
   image: ubuntu
   links:
     - db_2:mysql
   ports:
    - 80

db2:
  image: ubuntu
  environment:
    MYSQL_ROOT_PASSWORD: 123

It's also possible to combine several yml files like
$docker-compose -f [File A].yml -f [File B].yml up -d

2) Every container in the build can be controlled separately with compose.
$docker-compose stop/start/build/ ubuntu_1

3) Using $docker-compose build it will only rebuild where changes have been done.

Here is more information that could be useful https://docs.docker.com/compose/extends/#extending-services

If none of above is correct please example of build.

I edited my question and added an example. For using your solutions, we can write a makefile which parses each docker-compose.yml and combine them altogether in a single file. But this approach needs parsing the files and handle many error situations (like same db name in the example) which is kind of dirty. If we use -f option to run different yml files, we can't link services inside them together. I didn't catch how extends can help me in this situation. From the example in the doc it seems it helps to reuse configures from the same service. — Mehran Akhavan, Jan 31 '16 at 14:02
can't you just use db_a: / db_b: and you will have separate containers. — Mathias Asberg, Jan 31 '16 at 14:12

score 1 · Answer 3 · answered Mar 29 '16 at 10:12

This is our approach for anyone else having same problem:

Now each of our projects has a docker-compose which can be run standalone. We have another project called 'development-kit' which clones needed projects and store them in a directory. We can run our projects using command similiar to:

python controller.py --run projectA projectB

It runs each project using docker-compose up command. Then when all projects are up and running, it starts adding all other projects main docker's IP to other projects by adding them to the /etc/hosts ips using these commands:

# getting contaier id of projectA and projectB
CIDA = commands.getoutput("docker-compose ps -q %s" % projectA)
CIDB = commands.getoutput("docker-compose ps -q %s" % projectB)
# getting ip of container projectA
IPA = commands.getoutput("docker inspect --format '{{ .NetworkSettings.IPAddress }}' %s" % CIDA)

Now for sending requests from projectB to projectA we only need to define projectA IP as "projectA" in projectB's settings.

Running multiple projects using docker which each runs with docker-compose

3 Answers3

foo/docker-compose.yml:

bar/docker-compose.yml:

Linked