12

I have a dockerized application with a few services running using docker-compose. I'd like to connect this application with ElasticSearch/Logstash/Kibana (ELK) using another docker-compose application, docker-elk. Both of them are running in the same docker machine in development. In production, that will probably not be the case.

How can I configure my application's docker-compose.yml to link to the ELK stack?

Dag Høidahl
  • 7,873
  • 8
  • 53
  • 66
  • You can't. you either need to use your host machine network to connect to apps or use overlay networking with you docker compose which I believe still not mature enough for production use cases. – Boynux Dec 21 '15 at 17:37

5 Answers5

9

Update Jun 2016

The answer below is outdated starting with docker 1.10. See this other similar answer for the new solution. https://stackoverflow.com/a/34476794/1556338

Old answer

  1. Create a network:

     $ docker network create --driver bridge my-net
    
  2. Reference that network as an environment variable (${NETWORK})in the docker-compose.yml files. Eg:

    pg:
      image: postgres:9.4.4
      container_name: pg
      net: ${NETWORK}
      ports:
        - "5432"
    myapp:
      image: quay.io/myco/myapp
      container_name: myapp
      environment:
        DATABASE_URL: "http://pg:5432"
      net: ${NETWORK}
      ports:
        - "3000:3000"
    

Note that pg in http://pg:5432 will resolve to the ip address of the pg service (container). No need to hardcode ip addresses; An entry for pg is automatically added to the /etc/host of the myapp container.

  1. Call docker-compose, passing it the network you created:

    $ NETWORK=my-net docker-compose up -d -f docker-compose.yml -f other-compose.yml
    

I've created a bridge network above which only works within one node (host). Good for dev. If you need to get two nodes to talk to each other, you need to create an overlay network. Same principle though. You pass the network name to the docker-compose up command.

Community
  • 1
  • 1
Bernard
  • 16,149
  • 12
  • 63
  • 66
9

You could also create a network with docker outside your docker-compose :

docker network create my-shared-network 

And in your docker-compose.yml :

version: '2'
services:
  pg:
    image: postgres:9.4.4
    container_name: pg
    expose:
      - "5432"
networks:
  default:
    external:
      name: my-shared-network

And in your second docker-compose.yml :

version: '2'
  myapp:
    image: quay.io/myco/myapp
    container_name: myapp
    environment:
      DATABASE_URL: "http://pg:5432"
    net: ${NETWORK}
    expose:
      - "3000"
networks:
  default:
    external:
      name: my-shared-network

And both instances will be able to see each other, without open ports on host, you just need to expose ports, and there will see each other through the network : "my-shared-network".

Antoine
  • 4,456
  • 4
  • 44
  • 51
2

If you set a predictable project name for the first composition you can use external_links to reference external containers by name from a different compose file.

In the next docker-compose release (1.6) you will be able to use user defined networks, and have both compositions join the same network.

dnephin
  • 25,944
  • 9
  • 55
  • 45
1

Take a look at multi-host docker networking

Networking is a feature of Docker Engine that allows you to create virtual networks and attach containers to them so you can create the network topology that is right for your application. The networked containers can even span multiple hosts, so you don’t have to worry about what host your container lands on. They seamlessly communicate with each other wherever they are – thus enabling true distributed applications.

Rob van Laarhoven
  • 8,737
  • 2
  • 31
  • 49
  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/low-quality-posts/10638334) – Mel Dec 21 '15 at 12:58
  • @tmoreau I agree. But this question is too broad, the topic too complex, to give a short answer. I removed the link so OP can google. – Rob van Laarhoven Dec 21 '15 at 13:10
  • If the question is too broad, flag as such. Your answer could be a comment. – Mel Dec 21 '15 at 13:11
  • 3
    That's a matter of opinion. The answer is too broad, not the question. Answers are for adding solutions to the question. Comments to add essential information, ask for clarification, ask for extra information, etc. Although short the answer may serve as a future reference. – Rob van Laarhoven Dec 21 '15 at 13:36
1

I didn't find any complete answer, so decided to explain it in a complete and simple way.

To connect two docker-compose you need a network and putting both docker-composes in that network, you could create netwrok with docker network create name-of-network,

or you could simply put network declaration in networks option of docker-compose file and when you run docker-compose (docker-compose up) the network would be created automatically.

put the below lines in both docker-compose files

networks:
   net-for-alpine:
     name: test-db-net

Note: net-for-alpine is internal name of the network and it will be used inside of the docker-compose files and could be different, test-db-net is external name of the network and must be same in two docker-compose files.

Assume we have docker-compose.db.yml and docker-compose.alpine.yml

docker-compose.apline.yml would be:

version: '3.8'

services:

  alpine:
    image: alpine:3.14
    container_name: alpine
    networks:
      - net-for-alpine
  
    # these two command keeps apline container running
    stdin_open: true # docker run -i
    tty: true # docker run -t



networks:
  net-for-alpine:
    name: test-db-net

docker-compose.db.yml would be:

version: '3.8'

services:

  db:
    image: postgres:13.4-alpine
    container_name: psql
    networks:
      - net-for-db
  
networks:
  net-for-db:
    name: test-db-net

To test the network, go inside alpine container

docker exec -it alpine sh 
      

then with following commands you can check the network

# if it returns 0 or see nothing as a result, network is established
nc -z psql (container name)  

or

ping pgsql