docker stack: Redis not working on worker node - docker

I just completed the docker documentation and created two instances on aws (http://13.127.150.218, http://13.235.134.73). The first one is manager and the second one is the worker. Following is the composed file I used to deploy
version: "3"
services:
web:
# replace username/repo:tag with your name and image details
image: username/repo:tag
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "80:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
ports:
- "6379:6379"
volumes:
- "/home/docker/data:/data"
deploy:
placement:
constraints: [node.role == manager]
command: redis-server --appendonly yes
networks:
- webnet
networks:
webnet:
Here the redis service has the constraint that restricts it to run only on manager node. Now my question is how the web service on worker instance is supposed to use the redis service.

You need to use the hostname parameter in all container, so you can use this value to access services from worker or to access from worker the services on manager.
version: "3"
services:
web:
# replace username/repo:tag with your name and image details
image: username/repo:tag
hostname: "web"
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "80:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
hostname: "visualizer"
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
hostname: "redis"
ports:
- "6379:6379"
volumes:
- "/home/docker/data:/data"
deploy:
placement:
constraints: [node.role == manager]
command: redis-server --appendonly yes
networks:
- webnet
networks:
webnet:
In addictional if you use the portainer instead of visualizer you can control you SWARM stack with more options:
https://hub.docker.com/r/portainer/portainer
BR,
Carlos

Consider the stack file as per the below example -
Regardless of where it is placed manager|worker all the services in the stack file being on the same network can use the embedded DNS functionality which helps to resolve each service by the service name defined.
In this case the service web makes use of service redis by its service name.
Here is an example of the ping command able to resolve the service web from within the container associated with the redis service -
Read more about the Swarm Native Service Discovery to understand this.

Related

Docker swarm with reverse proxy, run requests based on request uri path to certain node

I have the following nodes with hostnames docker-php-pos-web-1,docker-php-pos-web-2,docker-php-pos-web-3,and docker-php-pos-web-4 in a docker swarm cluster with caddy proxy configured on distributed mode
I want requests with cron anywhere in the url path to run on docker-php-pos-web-4. An example request would be demo.phppointofsale.com/index.php/ecommerce/cron. If "cron" is not in the url, it would route as normal.
I want to avoid having 2 copies of production_php_point_of_sale_app just for this.
I am already routing to docker-php-pos-web-4 from my load balancer for "cron" in request path, BUT since in docker swarm the mesh network can decide on which node actually "runs" it. I always want docker-php-pos-web-4 to run these tasks
Below is my docker-compose.yml file
version: '3.9'
services:
production_php_point_of_sale_app:
logging:
driver: "local"
deploy:
restart_policy:
condition: any
mode: global
labels:
caddy: "http://*.phppointofsale.com, http://*.phppos.com"
caddy.reverse_proxy.trusted_proxies: "private_ranges"
caddy.reverse_proxy: "{{upstreams}}"
image: phppointofsale/production-app
build:
context: "production_php_point_of_sale_app"
restart: always
env_file:
- production_php_point_of_sale_app/.env
- .env
networks:
- app_network
- mail
caddy_server:
image: lucaslorentz/caddy-docker-proxy:ci-alpine
ports:
- 80:80
networks:
- caddy_controller
- app_network
environment:
- CADDY_DOCKER_MODE=server
- CADDY_CONTROLLER_NETWORK=10.200.200.0/24
volumes:
- caddy_data:/data
deploy:
restart_policy:
condition: any
mode: global
labels:
caddy_controlled_server:
caddy_controller:
image: lucaslorentz/caddy-docker-proxy:ci-alpine
networks:
- caddy_controller
- app_network
environment:
- CADDY_DOCKER_MODE=controller
- CADDY_CONTROLLER_NETWORK=10.200.200.0/24
volumes:
- /var/run/docker.sock:/var/run/docker.sock
deploy:
restart_policy:
condition: any
placement:
constraints: [node.role == manager]
networks:
caddy_controller:
driver: overlay
ipam:
driver: default
config:
- subnet: "10.200.200.0/24"
app_network:
driver: overlay
mail:
driver: overlay
volumes:
caddy_data: {}

docker swarm list dependencies of a service

Let's say we have the following stack file:
version: "3"
services:
ubuntu:
image: ubuntu
deploy:
replicas: 2
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
entrypoint:
- tail
- -f
- /dev/null
logging:
driver: "json-file"
ports:
- "80:80"
networks:
- webnet
web:
image: httpd
ports:
- "8080:8080"
hostname: "apache"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
resources:
limits:
memory: 32M
reservations:
memory: 16M
depends_on:
- "ubuntu"
networks:
- webnet
networks:
webnet:
When I run docker service inspect mystack_web the output generated does not show any reference to the depends_on entry.
Is that okay? and how can I print the dependencies of a given docker service?
The depends_on isn't used on docker swarm:
The depends_on option is ignored when deploying a stack in swarm mode with a version 3 compose file. - from Docker Docs
Another good explanation on GitHub:
depends_on is a no-op when used with docker stack deploy. Swarm mode services are restarted when they fail, so there's no reason to delay their startup. Even if they fail a few times, they will eventually recover. - from GitHub

Docker Compose: Cannot connect to Redis

I'm following the Docker Compose tutorial here
https://docs.docker.com/get-started/part5/#recap-optional
version: "3"
services:
web:
image: example/get-started:part-1
deploy:
replicas: 10
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "80:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
ports:
- "6379:6379"
volumes:
- ./data:/data
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
driver:
build: .
links:
- redis
networks:
webnet:
and while Redis seems to be running on myvm1, the app is unable to connect to it, and gives an error.
This is the app code in case it matters:
from flask import Flask
from redis import Redis, RedisError
import os
import socket
redis = Redis(host="redis", db=0, socket_connect_timeout=0, socket_timeout=0)
app = Flask(__name__)
#app.route("/")
def hello():
try:
visits = redis.incr("counter")
except RedisError:
visits = "<i>cannot connect to redis. Counter disabled</i>"
html = "<h3>Hello {name}!</h3>" \
"<b>Hostname:</b> {hostname}<br/>" \
"<b>Visits:</b> {visits}"
return html.format(name=os.getenv("NAME", "World"), hostname=socket.gethostname(), visits=visits)
if __name__ == "__main__":
app.run(host='0.0.0.0', port=80)
VM IPs:
myvm1 - virtualbox Running tcp://192.168.99.101:2376
v17.07.0-ce
myvm2 - virtualbox Running tcp://192.168.99.102:2376
v17.07.0-ce
Redis is running without errors on VM.
Any idea? There are many similar discussions online, but none helped yet.
If the redis is running on the VM , the binding might not be proper . Can you please check if its binding on 0.0.0.0 or else you need to edit the redis
config to bind on 0.0.0.0 and port for external service to connect to it

Docker Stack Swarm - Service Replicas are not spread for Mutli Service Stack

I have deployed a stack with a of 4 services on two hosts (docker compose version 3).
The services are Elasticsearch, Kibana. Redis, Visualiser and finally my Web App. I have't set any resource restrictions yet.
I spun two virtual host via docker-machine , one with 2GB and one with 1GB.
Then I increased the replicas of my web app to 2 replicas, which resolved to the following distribution:
Host1 (Master):
Kibana, Redis, Web App, Visualiser, WebApp
Host2 (Worker):
Elasticsearch
Why is the Swarm Manager distributing both Web App Containers to the same host. Wouldn't it be smarter if Web App is distributed to both hosts?
Besides node tagging I couldn't find any other way in the docs to influence the distribution.
Am I missing something?
Thanks
Bjorn
docker-compose.yml
version: "3"
services:
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.4.3
environment:
ES_JAVA_OPTS: -Xms1g -Xmx1g
ulimits:
memlock: -1
nofile:
hard: 65536
soft: 65536
nproc: 65538
deploy:
resources:
limits:
cpus: "0.5"
memory: 1g
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
networks:
- webnet
web:
# replace username/repo:tag with your name and image details
image: bjng/workinseason:swarm
deploy:
replicas: 2
restart_policy:
condition: on-failure
ports:
- "80:6000"
networks:
- webnet
kibana:
image: docker.elastic.co/kibana/kibana:5.4.3
deploy:
placement:
constraints: [node.role == manager]
ports:
- "5601:5601"
networks:
- webnet
redis:
image: "redis:alpine"
networks:
- webnet
volumes:
esdata:
driver: local
networks:
webnet:
Docker schedules tasks (containers) based on available resources; if two nodes have enough resources, the container can be scheduled on either one.
Recent versions of Docker use "HA" scheduling by default, which means that tasks for the same service are spread over multiple nodes, if possible (see this pull request) https://github.com/docker/swarmkit/pull/1446

Docker Swarm connection between containers refused for some containers

simplified swarm:
manager1 node
- consul-agent
worker1 node
- consul-client1
- web-app:80
- web-network:9000
database1 node
- consul-client2
- redis:6379
- mongo:27017
The web-app and web-network services can connect to redis and mongo through their service names correctly, e.g redis.createClient('6379', 'redis') and mongoose.connect('mongodb://mongo').
However, container web-app cannot connect to web-network, I'm trying to make a request like so:
request('http://web-network:9000')
But get the error:
errorno: ECONNREFUSED
address: 10.0.1.9
port: 9000
Request to web-network using a private IP does work:
request('http://11.22.33.44:9000')
What am I missing? Why can they connect to redis and mongo but not between each container? When moving redis/mongo to the same node as web-app, it will still work, so I don't think the issue comes because the services cannot talk to a service on the same server node.
Can we make docker network use private IP instead of the pre-configured subnet?
docker stack deploy file
version: '3'
services:
web-app:
image: private-repo/private-image
networks:
- swarm-network
ports:
- "80:8080"
deploy:
placement:
constraints:
- node.role==worker
web-network:
image: private-repo/private-image2
networks:
- swarm-network
ports:
- "9000:8080"
deploy:
placement:
constraints:
- node.role==worker
redis:
image: redis:latest
networks:
- swarm-network
ports:
- "6739:6739"
deploy:
placement:
constraints:
- engine.labels.purpose==database
mongo:
image: mongo:latest
networks:
- swarm-network
ports:
- "27017:27017"
deploy:
placement:
constraints:
- engine.labels.purpose==database
networks:
swarm-network:
driver: overlay
docker stack deploy app -c docker-compose.yml

Resources