docker container seems to have failing /etc/hosts until an _interactive_ docker start

Question

If I "docker-compose start service" in an interactive terminal (not -it, but a bash running on the host interactively), my service initializes happily.

But if I put that same command in a shell script, it fails.

I suspect the problem is name resolution via /etc/hosts; it seems to work interactively but not in a script for some reason. I want the containers to be able to see the names defined in the host's /etc/hosts. And although I am on Linux Mint 19.1 I need it to work on Windows, Mac and Linux.

Things I've tried that didn't work:

Adding sleeps - as long as 10 minutes
Checking for environment variable differences
Running the command under tcsh instead of bash
Running the command in a pseudo-tty using /usr/bin/script -c

But if I throw a /bin/bash -i in my script, with an echo that says what command to type, it works!

I'm using:

docker version 18.09.6, build 481bc77
docker-compose version 1.24.0, build 0aa59064

I'd have to get an OK from my management to share more than small snippets of code.

I want the service to start via docker-compose up -d or at least docker-compose start - in a script, for the sake of automation.

The error message inside the container (from docker logs -f service) looks like: elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host=u'elasticsearch', port=9200): Read timed out. (read timeout=10))

I sometimes see a high load in elasticsearch, which seems a bit odd. By high I mean as high as 490% on an 8 core (probably counting hyperthreading). More commonly it's down around 5-15%.

Here's an SSCCE:

#!/bin/bash

set -eu
set -o pipefail
set -x

docker stop elasticsearch || true
docker-compose start elasticsearch

# Give elasticsearch some time to come up.
# Normally I use a small REST client that calls ES until it starts responding - but that wouldn't be self-contained.
sleep 120

docker stop service || true
docker-compose start service

The script itself runs to completion fine, but then "service" exits before it should.

Thanks!

The script and the exact error message might help others to help you. If you cannot share a specific code, there is almost always a solution [to create a minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) — Zeitounator, May 29 '19 at 15:47
I am not sure I understand the problem correctly but I think what you want to use is `docker-compose up -d elasticsearch` instead of `docker-compose start elasticsearch` — user7440787, May 30 '19 at 14:40
I think this issue is related to this [one](https://stackoverflow.com/questions/31746182/docker-compose-wait-for-container-x-before-starting-y) — user7440787, May 30 '19 at 14:48
@user7440787 I use "up" on the first startup of the container, and "start' on subsequent startups of the container. — dstromberg, May 30 '19 at 16:21
@user7440787 Question 31746182 does look somewhat relevant, though I suspect I need to wait for something Inside the container - specifically for the pseudo-DNS to come up, so that it can resolve things in the host's /etc/hosts. — dstromberg, May 30 '19 at 16:59

score 0 · Answer 1 · answered May 31 '19 at 08:12

If I understood correctly, you want the elasticsearch container to serve your service container (i.e. both container to contact each other) hence, the service container needs to know the elasticsearch's IP address.

In that case, docker-compose will manage the containers addresses, what you will need to do is to store that information in the service container as an environment variable. The plus side of it, is that you can start both container with the same command (just docker-compose up -d). For example:

services:
  elasticsearch:
      image: YOUR_ELASTICSEARCH_IMAGE
      ports:
        - 9200:9200
  service:
      image: YOUR_SERVICE_IMAGE
      environment:
          - ELASTICSEARCH_HOST=elasticsearch

If there is a dependency between containers and the elasticsearch container needs some time to startup, then you can either put a sleep statement in the service at startup or use the solution in 31746182

Yes, "service" depends on elasticsearch. With regard to environment variables: That's something we'll have to think about. We have our development systems on which we have elasticsearch on 127.0.0.1, and our production environments where elasticsearch is something else. So if we can set different environment variable values for the different systems, that might work - especially if it does mean changing our docker yaml frequently. — dstromberg, May 31 '19 at 16:47
It's still strange that sometimes name resolution works, and sometimes it doesn't. :-S — dstromberg, May 31 '19 at 16:47

score 0 · Accepted Answer · answered Jun 07 '19 at 20:42

It appears that docker-compose sometimes really wants to be run on a pty instead of from a script directly/normally. I suppose that could somehow be related to the content of the containers I'm starting, and not docker itself.

Anyway, believe it or not, running my up's like (for example) :

script -q -c "docker-compose up -d \"elasticsearch\"" /dev/null

...solved the problem.

Weird, I know, but I'm not currently planning to figure out why. I'm just happy it's working.

docker container seems to have failing /etc/hosts until an _interactive_ docker start

2 Answers2