Docker swarm. Unable to run exec against a container of a docker stack, because "docker container list" does not find the container

Question

Section 1

I am trying to execute "exec" on one of the containers in a stack or service. I have followed the answers provided here execute a command within docker swarm service as well as on the official Docker documentation here https://docs.docker.com/engine/swarm/secrets/ but it seems "docker container list" and its variants ("docker ps") seem not to find any container listed in the "docker service list".

See the example below from https://docs.docker.com/engine/swarm/secrets/ which is meant to use of "exec" on a container of a service.

1)

printf "This is a secret" | docker secret create my_secret_data -

docker service  create --name redis --secret my_secret_data redis:alpine

docker service ps redis

yields ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 75olesuh9n4h redis.1 redis:alpine virt-05 Running Running 16 minutes ago

4. The two commands below return zero records for "redis"

docker ps --filter name=redis -q
docker container list

The two commands below return no records for "redis" hence the commands below are bound not to work.

docker container exec $(docker ps --filter name=redis -q) ls -l /run/secrets
docker container exec $(docker ps --filter name=redis -q) cat /run/secrets/my_secret_data

What could be different in my environment to make it not possible to reproduce the results in the example above (https://docs.docker.com/engine/swarm/secrets/)

My environment is as follows

$ uname -r
4.18.0-338.el8.x86_64

$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:53:39 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:00 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Section 2

I have just initialized a docker swam on another node and I am able to successfully reproduce the behaviour the example https://docs.docker.com/engine/swarm/secrets/.

In this second environment docker ps is able to find (list) the redis container started as service. The command below returns a record.

docker ps --filter name=redis -q
3de724329171

I was able to successfully run "exec" on this container by running the commands below

docker container exec $(docker ps --filter name=redis -q) ls -l /run/secrets
docker container exec $(docker ps --filter name=redis -q) cat /run/secrets/my_secret_data

The docker version of if this second environment is given below

$ uname -r
3.10.0-1160.36.2.el7.x86_64

docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:49 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:54:13 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Answer 1

To use docker to access containers in multi node swarm, to avoid having to log into each vm / docker host individually you are going to want to ensure your docker swarm nodes are setup for remote access.

The docker daemon has parameters to expose a public socket :2375 for insecure, or :2376 for mtls protected comms. mtls is recommended for anything but the most sandboxed lab environments. Alternatively, passwordless ssh access (where you register your local .ssh_id on each of the servers) allows you to use ssh urls to access docker remotely and is the recommended option.

Once you have enabled either ssh or tcp access to docker then you can pass the remote node url to docker, set it in an env variable, or use it via a docker context:

# using DOCKER_HOST
export DOCKER_HOST=tcp://lab-node1:2375
docker container ls
# Passing a url directly
docker -H ssh://user@lab-node2 container ls
# Using a docker context
docker context create node1 --docker "host=ssh://user@node1"
docker context use node1
docker container ls

Once you have multi docker access, ensure that docker is pointing at a manager node and get the relevant details of the service:

In this case, we need to know the node that service tasks are running on, and the container name of the tasks, which is a combination of the name and id.

docker service ps your-service --format '{{.Node}} {{.Name}}.{{.ID}}' --no-trunc --filter desired-state=running

Now that you have the remote node, assuming you have created contexts for each node, then a command like this (where $containername is the '{{.Name}}.{{.ID}}' string.

docker -c $node exec $containername cmdline

Answer 2

Solved (credit Chris Becke)

I needed to try accessing the container of the service from node the container is running.

I credit Chris Becke for this information.

The swarm setup in Section 1 was a multi-node setup while the swarm setup in Section 2 is a single node (manager/worker) setup. The container in section 1 was dispatched to a worker node other than the one I was trying to give exec commands to.
I was able to successfully run the "exec" command on the service container once I logged into the worker node.