7

Let me first describe my original problem, and solution:

I have several docker-compose files which describe different parts of my application. The parts are developed and deployed independently, so they can not be integrated into a single compose file. But those components need to communicate with each other, and the solution I am using at the moment is to have an external network (bridge) which all services are connecting to. So far so good, and I can indeed connect to my service started with any docker compose file as long as I am connected to the custom bridge network:

$ docker run --network=mynet --rm --name ping_test -it xenial-networking bash

root@0319469f7951:/# ping -c 1 proj_web_1
PING proj_web_1 (172.30.0.3) 56(84) bytes of data.
64 bytes from 172.30.0.3: icmp_seq=1 ttl=64 time=0.071 ms

--- proj_web_1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.071/0.071/0.071/0.000 ms

Why am using proj_web_1? Because that is the DNS entry that docker compose is creating.

This works fine, and nobody cares that underscores are not that great in domain names.

Nobody ... except django it seems:

ERROR Invalid HTTP_HOST header: 'proj_web_1:8000'. The domain name provided is not valid according to RFC 1034/1035.
172.30.0.5 - admin [07/May/2018:05:41:53 +0000] "OPTIONS /api/v1/expansions/ HTTP/1.1" 400 58663 "-" "python-requests/2.18.4"

It seems that docker-compose does not support using hyphens instead of underscores.

It could be that I can workaround this by enabling DEBUG in my django service.

Is there a cleaner way of enabling this, without running in django debug mode? Is there a way to tell docker compose to stop using underscores?

blueFast
  • 41,341
  • 63
  • 198
  • 344
  • I can't reproduce this. `requests.post('http://proj_web1:8000/api/v1/expansions/')` does not complain about the host name, but results in a timeout as expected on my machine. Please provide an MCVE! – Klaus D. May 07 '18 at 06:19
  • Also see: https://code.djangoproject.com/ticket/18517 – Timir May 07 '18 at 06:26
  • Note that docker compose also allows you to use the service name (so just `web` in your case) to communicate between services in the same stack (which is also the recommendation, as it doesn't make them dependent on the project name used to deploy the stack) – thaJeztah May 07 '18 at 06:29
  • @KlausD. the problem occurs on the receiving side. You need to have a django-rest based service to see that problem. – blueFast May 07 '18 at 06:31
  • @thaJeztah not possible when using different docker compose: you can not reach services declared by one docker compose from containers started with another docker compose by just using the service name. – blueFast May 07 '18 at 06:32
  • @Timir yes, that is part of the problem. But I am not sure django is to blame, since underscores are really not great in urls. – blueFast May 07 '18 at 06:34
  • Related: https://github.com/docker/compose/issues/472#issuecomment-63514093 – Sraw May 07 '18 at 06:40
  • A hostname in the DNS, and hence a domain name, can not have underscores. This is per the DNS standard, see RFC1034 (section 3.5). – Patrick Mevzek May 07 '18 at 13:20
  • @PatrickMevzek The RFCs can say all they want, docker compose is using underscores by default. – blueFast May 08 '18 at 08:23
  • @dangonfast while the short aliases are intended to be used in the same stack, they are done through [network-scoped aliases](https://docs.docker.com/compose/compose-file/#aliases). Containers from another docker compose can use those aliases if they're connected to the same network. Here's a simple example: https://gist.github.com/thaJeztah/e4f6adff5bb1d9678557611ea1ff5afc – thaJeztah May 08 '18 at 10:59
  • @thaJeztah that is what I was looking for - and similar to what I mention in my answer below, but using builtin capabilities (no need to define an alias). I could accept that as an answer. – blueFast May 08 '18 at 12:36
  • @dragonfast you and docker can say all they want, the Internet works today because its interoperability is guaranteed by people following RFCs.Why are you trying to avoid underscores if everything works so well with them as you say? – Patrick Mevzek May 08 '18 at 13:33
  • @dragonfast posted my gist as an answer – thaJeztah May 08 '18 at 13:41

2 Answers2

6

Cross-compose network connections

When running a compose project, services are accessible both through their full name (including the project-name prefix, for example, myproject_web_1), and through their service name (as specified in the compose-file), for example web. The short name is a network-scoped alias, which means that any container connected to the same network can access the container through this name.

By default, docker-compose creates a network for each compose project (projectname_default) so that all services in the compose project can communicate. Because that network is created for each project individually, two compose project don't share the same network, and their services are isolated from other compose projects.

It's possible, however, to make compose projects (or individual services in a compose project) share the same network.

Example 1 - using a shared network for the whole project

The following compose files specify a custom name for the default network; both compose-files use the sharednet network as the default, which means that services for both compose project will be connected to the same network:

Compose file 1 (compose1.yml):

version: '3.5'
services:
  compose1service:
    image: busybox
    tty: true

networks:
  default:
    name: sharednet

Compose file 2 (compose2.yml):

version: '3.5'
services:
  compose2service:
    image: busybox
    tty: true

networks:
  default:
    name: sharednet

To illustrate this:

Start both compose files:

docker-compose -f compose1.yml --project-name=compose1 up -d
docker-compose -f compose2.yml --project-name=compose2 up -d

Using the short (compose2service) name to ping the compose2service from inside the service in the compose1service container works;

docker exec compose1_compose1service_1 ping -c1 compose2service
PING compose2service (172.20.0.3): 56 data bytes
64 bytes from 172.20.0.3: seq=0 ttl=64 time=0.134 ms

--- compose2service ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.134/0.134/0.134 ms

And vice-versa:

docker exec compose2_compose2service_1 ping -c1 compose1service
PING compose1service (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: seq=0 ttl=64 time=0.151 ms

--- compose1service ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.151/0.151/0.151 ms

As does the full name (including the project prefix);

docker exec compose2_compose2service_1 ping -c1 compose1_compose1service_1
PING compose1_compose1service_1 (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: seq=0 ttl=64 time=0.151 ms

--- compose1_compose1service_1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.151/0.151/0.151 ms

Example 2 - using a shared network for some service in the project

In this example, both compose projects have their own ("default") network, but in addition, a shared network is added to allow some services to connect to services in the other compose project.

Compose file 1 (compose3.yml):

version: '3.5'
services:
  compose3service:
    image: busybox
    tty: true
    networks:
      - default
      - sharednet

  compose3otherservice:
    image: busybox
    tty: true

networks:
  sharednet:
    name: mysharednetwork

Note: if you specify which networks a service should be connected to, you override the defaults, which means that the service is no longer automatically connected to the default network. Include the default network in the list of networks to allow the service to communicate with other services in the compose project.

Compose file 2 (compose4.yml):

version: '3.5'
services:
  compose4service:
    image: busybox
    tty: true
    networks:
      - default
      - sharednet

  compose4otherservice:
    image: busybox
    tty: true

networks:
  sharednet:
    name: mysharednetwork

Start both compose files:

docker-compose -f compose3.yml --project-name=compose3 up -d
docker-compose -f compose4.yml --project-name=compose4 up -d

Again, pinging from the compose3service to the compose4service service in the other compose project works (either by short name or full name);

docker exec compose3_compose3service_1 ping -c1 compose4service
PING compose4service (172.22.0.3): 56 data bytes
64 bytes from 172.22.0.3: seq=0 ttl=64 time=0.110 ms

--- compose4service ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.110/0.110/0.110 ms


docker exec compose3_compose3service_1 ping -c1 compose4_compose4service_1
PING compose4_compose4service_1 (172.22.0.3): 56 data bytes
64 bytes from 172.22.0.3: seq=0 ttl=64 time=0.093 ms

--- compose4_compose4service_1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.093/0.093/0.093 ms

However, trying the same to connect to the compose4otherservice will fail, because that service is not connected to the mysharednetwork network:

docker exec compose3_compose3service_1 ping -c1 compose4otherservice
ping: bad address 'compose4otherservice'

docker exec compose3_compose3service_1 ping -c1 compose4_compose4otherservice_1
ping: bad address 'compose4_compose4otherservice_1'

Note:

Care should be taken if two compose projects have a service with the same name (for example, both have a service named web). In such situations, web may at random resolve to the web service from either project.

thaJeztah
  • 27,738
  • 9
  • 73
  • 92
1

I am able to workaround the problem by using a network alias, avoiding the underscore:

  web:
    restart: always
    environment:
      - DJANGO_SECRET_KEY=local
      - DJANGO_CONFIGURATION=Develop
    build:
      context: ./
      args:
        - REGISTRY
    image: web
    command: ./run-gunicorn.sh
    volumes:
      - ./:/code
    depends_on:
      - postgres
    networks:
      spp:
        aliases:
          - projweb

And now I can reach it:

root@a9c0fde612a1:/# ping -c 1 projweb
PING projweb (172.30.0.4) 56(84) bytes of data.
64 bytes from svc_web_1.spp (172.30.0.4): icmp_seq=1 ttl=64 time=0.082 ms

--- projweb ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.082/0.082/0.082/0.000 ms
blueFast
  • 41,341
  • 63
  • 198
  • 344