Why docker stack deploy is not starting all my containers? - docker

Question

I deployed my docker stack:
⇒ docker stack deploy -c docker-compose.yml my_stack
Creating network my_stack_network
Creating service my_stack_redis
Creating service my_stack_wsgi
Creating service my_stack_nodejs
Creating service my_stack_nginx
Creating service my_stack_haproxy
Creating service my_stack_postgres
But when I do docker container ls, it only shows three containers:
~|⇒ docker container ls | grep my_stack
212720bfafc3 postgres:11 "docker-entrypoint.s…" 4 minutes ago Up 3 minutes 5432/tcp my_stack_postgres.1.9nx7jb21whi61aboe9hmet6m2
3132dd980589 isiq/nginx-brotli:1.21.0 "/docker-entrypoint.…" 4 minutes ago Up 4 minutes 80/tcp my_stack_nginx.1.isl2c78z6w5ptizurm3a4cnte
62ef3c76fb9e redis:6.2.4 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 6379/tcp my_stack_redis.1.xnisrd1i6hod6jkm64623cpzj
But docker stack ps lists all of them as Running:
~|⇒ docker stack ps --no-trunc my_stack
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
1fqwlgblhi5q0cdl5cy75ucli my_stack_haproxy.1 haproxy:2.3.9#sha256:f63aabf39efcd277b04a503d38e59e80224a0c11f47b2568b13b0092698c5a3a Running New 2 minutes ago
isl2c78z6w5ptizurm3a4cnte my_stack_nginx.1 isiq/nginx-brotli:1.21.0#sha256:436cbc0d8cd051e7bdb197d7915fe90fa5a1bdadea6d02272ba117fccf30c936 tadoba Running Running 2 minutes ago
1myvtgl11qqw2xa9cv79uikcs my_stack_nodejs.1 nodejs:my_stack Running New 2 minutes ago
9nx7jb21whi61aboe9hmet6m2 my_stack_postgres.1 postgres:11#sha256:5d2aa4a7b5f9bdadeddcf87cf7f90a176737a02a30d917de4ab2e6a329bd2d45 tadoba Running Running 2 minutes ago
xnisrd1i6hod6jkm64623cpzj my_stack_redis.1 redis:6.2.4#sha256:6bc98f513258e0c17bd150a7a26f38a8ce3e7d584f0c451cf31df70d461a200a tadoba Running Running 2 minutes ago
mzmmb7a3bxjpfkfa3ea5o5w85 my_stack_wsgi.1 wsgi:my_stack Running New 2 minutes ago
Checking logs of containers not listed in docker container ls gives No such container error:
~|⇒ docker logs -f 1myvtgl11qqw2xa9cv79uikcs
Error: No such container: 1myvtgl11qqw2xa9cv79uikcs
~|⇒ docker logs -f mzmmb7a3bxjpfkfa3ea5o5w85
Error: No such container: mzmmb7a3bxjpfkfa3ea5o5w85
~|⇒ docker logs -f 1fqwlgblhi5q0cdl5cy75ucli
Error: No such container: 1fqwlgblhi5q0cdl5cy75ucli
What could be the reason? How can I debug this?
Update
It seems that services without dependencies are able to join the network. But services with dependencies on other services are not able to. I am unable to figure out the reason. Here is the gist with output of docker inspect network's output.
PS
I run watch 'docker container ls | grep my_app' in another terminal before running docker stack deploy .... But those three containers never appear in the watch list. Rest of three do appear.
I am running all nodes on the same remote machine connected through ssh. This is the output of docker node ls:
~|⇒ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
z9hovq8ry6qont3m2rbn6upy4 * tadoba Ready Active Leader 20.10.11
Here is my docker compose file for reference:
version: "3.8"
services:
postgres:
image: postgres:11
volumes:
- my_app_postgres_volume:/var/lib/postgresql/data
- type: tmpfs
target: /dev/shm
tmpfs:
size: 536870912 # 512MB
environment:
POSTGRES_DB: my_app_db
POSTGRES_USER: my_app
POSTGRES_PASSWORD: my_app123
networks:
- my_app_network
redis:
image: redis:6.2.4
volumes:
- my_app_redis_volume:/data
networks:
- my_app_network
wsgi:
image: wsgi:my_app3_stats
volumes:
- /my_app/frontend/static/
- ./wsgi/my_app:/my_app
- /my_app/frontend/clientApp/node_modules
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
depends_on:
- postgres
- redis
ports:
- 9090
environment:
C_FORCE_ROOT: 'true'
SERVICE_PORTS: 9090
networks:
- my_app_network
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
nodejs:
image: nodejs:my_app3_stats
volumes:
- ./nodejs/frontend:/frontend
- /frontend/node_modules
depends_on:
- wsgi
ports:
- 9998:9999
environment:
BACKEND_API_URL: http://aa.bb.cc.dd:9764/api/
networks:
- my_app_network
nginx:
image: isiq/nginx-brotli:1.21.0
volumes:
- ./nginx:/etc/nginx/conf.d:ro
- ./wsgi/my_app:/my_app:ro
- my_app_nginx_volume:/var/log/nginx/
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
networks:
- my_app_network
haproxy:
image: haproxy:2.3.9
volumes:
- ./haproxy:/usr/local/etc/haproxy/:ro
- /var/run/docker.sock:/var/run/docker.sock
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
depends_on:
- wsgi
- nodejs
- nginx
ports:
- 9764:80
networks:
- my_app_network
deploy:
placement:
constraints: [node.role == manager]
volumes:
my_app_postgres_volume:
my_app_redis_volume:
my_app_nginx_volume:
my_app_pgadmin_volume:
networks:
my_app_network:
driver: overlay
Output of docker service ps <service-name> for services not listed under tadoba node:
~/my_app|master-py3⚡
⇒ docker service ps my_app_nodejs
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
i04jpykp9ign my_app_nodejs.1 nodejs:bodhitree3_stats Running New about a minute ago
~/my_app|master-py3⚡
⇒ docker service ps my_app_haproxy
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
of4fcsxuq24c my_app_haproxy.1 haproxy:2.3.9 Running New about a minute ago
~/my_app|master-py3⚡
⇒ docker service ps my_app_wsgi
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
yt9nuhule39z my_app_wsgi.1 wsgi:bodhitree3_stats Running New 2 minutes ago

Answer 1

Firstly all of the services are running which is good. But docker container ls isn't cluster-aware i.e. it is showing the current containers running on this node. From the output docker stack ps --no-trunc my_stack I can see that there is another node labeled tadoba. So if you can log in to the other node you can see the running containers.
You can list the nodes on your cluster by running docker node ls.
If you want you can set up your docker contexts so you can change your docker context which will remove the need to log in and log out of the nodes. You can find more info here.