1

I deployed my docker stack:

⇒  docker stack deploy -c docker-compose.yml my_stack
Creating network my_stack_network
Creating service my_stack_redis
Creating service my_stack_wsgi
Creating service my_stack_nodejs
Creating service my_stack_nginx
Creating service my_stack_haproxy
Creating service my_stack_postgres

But when I do docker container ls, it only shows three containers:

~|⇒  docker container ls | grep my_stack                     
212720bfafc3   postgres:11                     "docker-entrypoint.s…"   4 minutes ago   Up 3 minutes   5432/tcp                                     my_stack_postgres.1.9nx7jb21whi61aboe9hmet6m2
3132dd980589   isiq/nginx-brotli:1.21.0        "/docker-entrypoint.…"   4 minutes ago   Up 4 minutes   80/tcp                                       my_stack_nginx.1.isl2c78z6w5ptizurm3a4cnte
62ef3c76fb9e   redis:6.2.4                     "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes   6379/tcp                                     my_stack_redis.1.xnisrd1i6hod6jkm64623cpzj

But docker stack ps lists all of them as Running:

~|⇒  docker stack ps --no-trunc my_stack
ID                          NAME                  IMAGE                                                                                                     NODE      DESIRED STATE   CURRENT STATE           ERROR     PORTS
1fqwlgblhi5q0cdl5cy75ucli   my_stack_haproxy.1    haproxy:2.3.9@sha256:f63aabf39efcd277b04a503d38e59e80224a0c11f47b2568b13b0092698c5a3a                               Running         New 2 minutes ago                 
isl2c78z6w5ptizurm3a4cnte   my_stack_nginx.1      isiq/nginx-brotli:1.21.0@sha256:436cbc0d8cd051e7bdb197d7915fe90fa5a1bdadea6d02272ba117fccf30c936          tadoba    Running         Running 2 minutes ago             
1myvtgl11qqw2xa9cv79uikcs   my_stack_nodejs.1     nodejs:my_stack                                                                                                     Running         New 2 minutes ago                 
9nx7jb21whi61aboe9hmet6m2   my_stack_postgres.1   postgres:11@sha256:5d2aa4a7b5f9bdadeddcf87cf7f90a176737a02a30d917de4ab2e6a329bd2d45                       tadoba    Running         Running 2 minutes ago             
xnisrd1i6hod6jkm64623cpzj   my_stack_redis.1      redis:6.2.4@sha256:6bc98f513258e0c17bd150a7a26f38a8ce3e7d584f0c451cf31df70d461a200a                       tadoba    Running         Running 2 minutes ago             
mzmmb7a3bxjpfkfa3ea5o5w85   my_stack_wsgi.1       wsgi:my_stack                                                                                                       Running         New 2 minutes ago

Checking logs of containers not listed in docker container ls gives No such container error:

~|⇒  docker logs -f 1myvtgl11qqw2xa9cv79uikcs            
Error: No such container: 1myvtgl11qqw2xa9cv79uikcs

~|⇒  docker logs -f mzmmb7a3bxjpfkfa3ea5o5w85                         
Error: No such container: mzmmb7a3bxjpfkfa3ea5o5w85

~|⇒  docker logs -f 1fqwlgblhi5q0cdl5cy75ucli                         
Error: No such container: 1fqwlgblhi5q0cdl5cy75ucli

What could be the reason? How can I debug this?

Update

It seems that services without dependencies are able to join the network. But services with dependencies on other services are not able to. I am unable to figure out the reason. Here is the gist with output of docker inspect network's output.

PS

I run watch 'docker container ls | grep my_app' in another terminal before running docker stack deploy .... But those three containers never appear in the watch list. Rest of three do appear.

I am running all nodes on the same remote machine connected through ssh. This is the output of docker node ls:

~|⇒  docker node ls                          
ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
z9hovq8ry6qont3m2rbn6upy4 *   tadoba     Ready     Active         Leader           20.10.11

Here is my docker compose file for reference:

version: "3.8"

services:
    postgres:
        image: postgres:11
        volumes:
            - my_app_postgres_volume:/var/lib/postgresql/data
            - type: tmpfs
              target: /dev/shm
              tmpfs:
                size: 536870912 # 512MB
        environment:
            POSTGRES_DB: my_app_db
            POSTGRES_USER: my_app
            POSTGRES_PASSWORD: my_app123
        networks:
            - my_app_network

    redis:
        image: redis:6.2.4
        volumes:
            - my_app_redis_volume:/data
        networks:
            - my_app_network

    wsgi:
        image: wsgi:my_app3_stats
        volumes:
            - /my_app/frontend/static/
            - ./wsgi/my_app:/my_app
            - /my_app/frontend/clientApp/node_modules
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        depends_on:
            - postgres
            - redis
        ports:
            - 9090
        environment:
            C_FORCE_ROOT: 'true'
            SERVICE_PORTS: 9090
        networks:
            - my_app_network
        deploy:
            replicas: 1
            update_config:
                parallelism: 1
                delay: 10s
            restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 120s

    nodejs:
        image: nodejs:my_app3_stats
        volumes:
            - ./nodejs/frontend:/frontend
            - /frontend/node_modules
        depends_on:
            - wsgi
        ports:
            - 9998:9999 
        environment:
            BACKEND_API_URL: http://aa.bb.cc.dd:9764/api/ 
        networks:
            - my_app_network

    nginx:
        image: isiq/nginx-brotli:1.21.0
        volumes:
            - ./nginx:/etc/nginx/conf.d:ro
            - ./wsgi/my_app:/my_app:ro
            - my_app_nginx_volume:/var/log/nginx/
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        networks:
            - my_app_network
            
    haproxy:
        image: haproxy:2.3.9
        volumes:
            - ./haproxy:/usr/local/etc/haproxy/:ro
            - /var/run/docker.sock:/var/run/docker.sock
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        depends_on:
            - wsgi
            - nodejs
            - nginx
        ports:
            - 9764:80
        networks:
            - my_app_network
        deploy:
            placement:
                constraints: [node.role == manager]

volumes:
    my_app_postgres_volume:
    my_app_redis_volume:
    my_app_nginx_volume:
    my_app_pgadmin_volume:

networks:
    my_app_network:
        driver: overlay

Output of docker service ps <service-name> for services not listed under tadoba node:

~/my_app|master-py3⚡ 
⇒  docker service ps my_app_nodejs
ID             NAME                   IMAGE                     NODE      DESIRED STATE   CURRENT STATE            ERROR     PORTS
i04jpykp9ign   my_app_nodejs.1        nodejs:bodhitree3_stats             Running         New about a minute ago             

~/my_app|master-py3⚡ 
⇒  docker service ps my_app_haproxy 
ID             NAME                    IMAGE           NODE      DESIRED STATE   CURRENT STATE            ERROR     PORTS
of4fcsxuq24c   my_app_haproxy.1        haproxy:2.3.9             Running         New about a minute ago             

~/my_app|master-py3⚡ 
⇒  docker service ps my_app_wsgi   
ID             NAME                 IMAGE                   NODE      DESIRED STATE   CURRENT STATE       ERROR     PORTS
yt9nuhule39z   my_app_wsgi.1        wsgi:bodhitree3_stats             Running         New 2 minutes ago
Rnj
  • 1,067
  • 1
  • 8
  • 23
  • Please provide the `docker inspect` output on the task id. E.g. `docker inspect i04jpykp9ign` – BMitch Sep 21 '22 at 13:37
  • [Here is the output](https://gist.github.com/ranjanmaria/dcca2fbbb1e42c9b44a06b75a5082da8) for `docker inspect` for `wsgi` continer. Its the first one to have dependency. nodejs and haproxy depend on wsgi. – Rnj Sep 21 '22 at 14:04

2 Answers2

2

Firstly all of the services are running which is good. But docker container ls isn't cluster-aware i.e. it is showing the current containers running on this node. From the output docker stack ps --no-trunc my_stack I can see that there is another node labeled tadoba. So if you can log in to the other node you can see the running containers.

You can list the nodes on your cluster by running docker node ls.

If you want you can set up your docker contexts so you can change your docker context which will remove the need to log in and log out of the nodes. You can find more info here.

KiruT
  • 56
  • 2
  • But I am running all of them on same remote machine connected through ssh. And this machine is callled tadoba. In fact I am doing all this through the zch command prompt which looks like `myusername@tadoba:~|⇒ `. I have added output of my `docker node ls` and my docker compose file at the end of original post. – Rnj Sep 21 '22 at 08:38
  • can you also add docker service ps output? If so do this for the service not deployed on tadoba node. – KiruT Sep 21 '22 at 08:58
  • Added at the end of the original post ... – Rnj Sep 21 '22 at 09:16
  • Another fact is, I run `watch 'docker container ls | grep my_app'` in another terminal before running `docker stack deploy ...`. But those three containers never appear in the watch list. Rest of three do appear. – Rnj Sep 21 '22 at 09:28
  • Okay, I noticed that the services that are stuck on the NEW state, which is they are on initialization, have dependencies on other services. So there might be connectivity issues among the services. Can you inspect the my_app_network overlay network? – KiruT Sep 21 '22 at 12:06
  • ohh!! I just ran `docker inspect my_app_network` and it says `Error: No such network`!!! Is this the issue? All containers including and after the first container `wsgi` with the dependency on others seems to not have started. Now guessing how do I fix this ideally and why this might have occurred. I tried changin `my_app_network` to `my_app_network2`. Still same behavior. Only three containers starting and getting `Error: No such network` for this new network too. – Rnj Sep 21 '22 at 12:46
  • Nope the network seem to have named as `{stack_name}{network_name}`. So in my case it was `my_app_my_app_network2`. [Here](https://gist.github.com/ranjanmaria/294e8d55cd4790169f160ba1f4840330) is the output of `docker network inspect my_app_my_app_network2` – Rnj Sep 21 '22 at 12:52
  • Okay lets try creating the overlay attachable network by running the command `docker network create -d overlay --attachable my_app_network3` Then edit docker compose by change the service networks to the new network my_app_network3 and finally modify the networks section at the end like in [here](https://gist.github.com/ranjanmaria/294e8d55cd4790169f160ba1f4840330?permalink_comment_id=4310475#gistcomment-4310475) – KiruT Sep 21 '22 at 13:09
  • Yeah got it replied on gist. Clearly those three nodes are not joining the network. – Rnj Sep 21 '22 at 13:14
  • Copy pasted two lines `networks:\n - my_app_network3 ` from the working services `redis` to the rest non working ones `nodejs`, `wsgi` and `haproxy`, in the hope that if these services are joining the network then copy pasting their network config will make other services too join the network. Those lines were visually same, but just wanted to avoid possiblity of some hidden character. But still no luck! – Rnj Sep 21 '22 at 13:25
  • Also all services NOT using volumes listed in `volumes:` section (at the bottom of docker compose) are failing to join network. Are they stuck in some reboot loop? If I recall it correct, this stack worked initially only once yesterday. – Rnj Sep 21 '22 at 13:51
  • `docker stack ps --no-trunc my_stack`, this command helped me resolve a similar issue. – babis21 Mar 08 '23 at 13:58
-1

If your Docker stack is running but no replicas are being created, and you don't see any errors when running the docker service ps <STACK_NAME> command, restarting Docker might help resolve the issue.

You can try the following steps:

Open a terminal or command prompt. Restart Docker using the appropriate command for your operating system:

  • On Linux (systemd-based systems like Ubuntu): 
    sudo systemctl restart docker
  • On macOS: 
    sudo service docker restart
  • On Windows (PowerShell): 
    Restart-Service docker

Wait for Docker to restart and then check if the replicas are created by running the docker service ps <STACK_NAME> command again. Restarting Docker can help resolve various issues related to container management and networking, and it's often a good first step when troubleshooting unexpected behavior in Docker.

If the problem persists after restarting Docker, there might be other factors causing the issue, such as incorrect configuration or resource limitations. In that case, it would be helpful to provide more details about your environment, the specific Docker stack configuration, and any relevant error messages or logs.

Hamidreza Vakilian
  • 730
  • 8
  • 10