docker service create never returns

Question

What would prevent docker service create ... command from completing? It never returns to the command prompt or reports any kind of error. Instead, it repeatedly switches between, "new", "assigned", "ready", and "starting".

Here's the command:

docker service create --name proxy -p 80:80 -p 443:443 -p 8080:8080 --network proxy -e MODE=swarm vfarcic/docker-flow-proxy

This is from "The DevOps 2.1 Toolkit - Docker Swarm", a book by Viktor Farcic, and I've come to a problem I can't get past...

I'm trying to run a reverse proxy service on a swarm of three nodes, all running locally on a LinuxMint system, and docker service create never returns to the command prompt. It sits there and loops between statuses and always says overall progress: 0 out of 1 tasks...

• This is on a cluster of three nodes all running locally, created with docker-machine with one manager and two workers.
• While testing, I've successfully created other overlay networks and other services using the same cluster of nodes.

As far as I can tell, the only significant differences between this and the other services I created for these nodes are:

1. There are three published ports. I double-checked the host and there's nothing else listening on any of those ports.
2. I don't have a clue what the vfarcic/docker-flow-proxy does.

I'm running this on a (real, not VirtualBox/VMWare/Hyper-V) Linux Mint 19 system (based on Ubuntu 18.04) with Docker 18.06. My host has 16Gb RAM and plenty of spare disk space. It was set up for the sole purpose of learning Docker after having too many problems on a Windows host.

Update:

If I Ctrl-C in the terminal where I tried to create the service, I get this message:

Operation continuing in background. Use docker service ps p7jfgfz8tqemrgbg1bn0d06yp to check progress.

If I use docker service ps --no-trunc p473obsbo9tgjom1fd61or5ap I get:

ID                          NAME                IMAGE                                                                                                         NODE                DESIRED STATE       CURRENT STATE               ERROR                                                          PORTS
dbzvr67uzg5fq9w64uh324dtn   proxy.1             dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Running             Starting 11 seconds ago
vzlttk2xdd4ynmr4lir4t38d5    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed 16 seconds ago       "task: non-zero exit (137): dockerexec: unhealthy container"
q5dlvb7xz04cwxv9hpr9w5f7l    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed 49 seconds ago       "task: non-zero exit (137): dockerexec: unhealthy container"
z86h0pcj21joryb3sj59mx2pq    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed about a minute ago   "task: non-zero exit (137): dockerexec: unhealthy container"
uj2axgfm1xxpdxrkp308m28ys    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed about a minute ago   "task: non-zero exit (137): dockerexec: unhealthy container"

So, no mention of node-2 or node-3 and I don't know what the error message means.

If I then use docker service logs -f proxy I get the following messages repeating every minute or so:

proxy.1.l86x3a6md766@node-1 | 2018/09/11 12:48:46 Starting HAProxy proxy.1.l86x3a6md766@node-1 | 2018/09/11 12:48:46 Getting certs from http://202.71.99.195:8080/v1/docker-flow-proxy/certs

I can only guess that it fails to get whatever certs it's looking for. There doesn't seem to be anything at that address, but an IP address lookup shows that it leads to my ISP. Where's that coming from?

Answer 1

There may be more than one motives why docker can't create the service. The simplest to see why is to see the last status of the service by running:

docker service ps --no-trunc proxy

where you have an ERROR column that describes the last error.

Alternatively you can run

docker service logs -f proxy

to see what's going on inside the proxy service.

Answer 2

This was an old question, but I was experimenting with docker and had this same issue. For me this issue was resolved by publishing the ports before stipulating the name.

I believe docker functions after stipulating the name as: --name "name" "image to use" "commands to pass to image on launch"

So it continues attempting to pass those options to the container as commands and fails instead of using them as options when running the service.

Answer 3

Add -t flag into command to prevent container(s) exit.

Answer 4

The same problem happened to me as well.

I think in my case, the image had some kind of error. I did the following experiments:

1. I first pulled the image, and created the service
   docker service create --name apache --replicas 5 -p 5000:80 [image repo]
   The same error from the question happened.
2. Like the first answer above, I tried a different order
   docker service create --replicas 5 -p 5000:80 --name apache [image repo]
   Same error occurred
3. I tried to create the service w/o the image downloaded locally, but the same error occurred.
4. I tried a different image, which worked before, then the service was created successfully.

The first image I used was swedemo/firstweb. (This is NOT my repo, it's my prof's)
I didn't examine what part of this image was the problem but hope this can give clue to others.

Answer 5

For me, it was some startup script issues causing the container to exit with 0 code. As my docker service has to have at least one replica (given as constraints), docker swarm try to recreate it. The container fail right after being relaunched by docker daemon hence causing the loop.