93

I often have problems because a service that I deploy on docker swarm with multiple nodes won't start and there are not logs generated that I can look at with docker service logs {serviceName}

There are many possible reasons for a service not to start such as

  • Can't download image from registry
  • Constraints that can't be fulfilled

I have trouble finding out why a container won't start. I found the command docker service ps {serviceName} which List the tasks of one or more services and a short error message (if there was an error). However when I try to inspect the task with docker service logs {taskId} (which should show logs of a task) I get Error response from daemon: task 3lkgo8t2sn7k not found.

Can anyone help me get a full error message why a service won't start?

herm
  • 14,613
  • 7
  • 41
  • 62

2 Answers2

214

I found one handy solution to the problem.

docker service ps --no-trunc {serviceName}

which will show errors with downloading images, mounting nfs volumes amongst others.

---------------------- UPDATE

Not all errors can be found in the way described above. Another usefull tool is looking at the docker deamon logs which can be done the follwing way as explained on stackoverflow: 

journalctl -u docker.service | tail -n 50

It depends on your OS. Here are the few locations, with commands for few Operating Systems:

  • Ubuntu (old using upstart ) - /var/log/upstart/docker.log
  • Ubuntu (new using systemd ) - journalctl -u docker.service
  • Boot2Docker - /var/log/docker.log
  • Debian GNU/Linux - /var/log/daemon.log
  • CentOS - /var/log/daemon.log | grep docker
  • CoreOS - journalctl -u docker.service
  • Fedora - journalctl -u docker.service
  • Red Hat Enterprise Linux Server - /var/log/messages | grep docker
  • OpenSuSE - journalctl -u docker.service
  • OSX - ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log/d‌​ocker.log
  • Windows - Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time, as mentioned here.
herm
  • 14,613
  • 7
  • 41
  • 62
  • 32
    There needs to be an easier way to get to this information with Swarm. It's much too cryptic. – Sarke Oct 01 '19 at 05:51
  • 1
    So basically there is no way to ensure (e.g. in CI/CD pipeline) that stack was deployed. – EugZol Mar 01 '21 at 18:03
  • https://stackoverflow.com/a/47206690/7683711 you could try building something similar to what is described in that link for docker swarm.... – herm Mar 05 '21 at 09:41
  • Here's an actual answer! Use `docker stack ps {stackname}` to get the `ID`, then run `docker inspect {task_id}` and you'll actually get a readout of issues under the "Status" key. https://stackoverflow.com/a/55093302/5728276 – user1847 Feb 16 '22 at 06:04
0

Another trick I've been using is from this article. I was quite stuck and this trick got me a bit further down the road:

docker run --network my-network -it --rm --entrypoint bash my-service

The problem is that when I ran it this way, I could start the service and interact with the other services in the swarm (via the external overlay network). But when I tried to start all three services with Docker Compose, the above service failed to start. So I'm still stuck, but slightly closer than before.

I often grep the syslog too, which sometimes reveals some nuggets:

less /var/log/syslog | grep docker
less /var/log/syslog | grep error

And, finally, here is a trick I found that sometimes works:

https://faun.pub/debug-docker-swarm-services-eec20fe3d13e

nicorellius
  • 3,715
  • 4
  • 48
  • 79