I'm quite new to Netdata and also Docker Swarm. I ran Netdata for a while on single hosts but now trying to stream Netdata from workers to a manager node in a swarm environment where the manager also should act as a central Netdata instance. I'm aiming to only monitor the data from the manager.
Here's my compose file for the stack:
version: '3.2'
services:
netdata-client:
image: titpetric/netdata
hostname: "{{.Node.Hostname}}"
cap_add:
- SYS_PTRACE
security_opt:
- apparmor:unconfined
environment:
- NETDATA_STREAM_DESTINATION=control:19999
- NETDATA_STREAM_API_KEY=1x214ch15h3at1289y
- PGID=999
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- netdata
deploy:
mode: global
placement:
constraints: [node.role == worker]
netdata-central:
image: titpetric/netdata
hostname: control
cap_add:
- SYS_PTRACE
security_opt:
- apparmor:unconfined
environment:
- NETDATA_API_KEY_ENABLE_1x214ch15h3at1289y=1
ports:
- '19999:19999'
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- netdata
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
networks:
netdata:
driver: overlay
attachable: true
Netdata on the manager works fine and the container runs on the one worker node I'm testing on. According to log output it seems to run well and gathers names from the docker containers running as it does in a local environment.
Problem is that it can't connect to the netdata-central service running on the manager.
This is the error message:
2019-01-04 08:35:28: netdata INFO : STREAM_SENDER[7] : STREAM 7 [send to control:19999]: connecting...,
2019-01-04 08:35:28: netdata ERROR : STREAM_SENDER[7] : Cannot resolve host 'control', port '19999': Name or service not known,
not sure why it can't resolve the hostname, thought it should work that way on the overlay network. Maybe there's a better way to connect and not rely on the hostname?
Any help is appreciated.
EDIT: as this question might come up - the firewall (ufw) on the control host is inactive, also I think the error message clearly points to a problem with name resolution.
Your API-Key is in the wrong format..it has to be a GUID. You can generate one with the "uuidgen" command...
https://github.com/netdata/netdata/blob/63c96aa96f96f3aea10bdcd2ecd92c889f26b3af/conf.d/stream.conf#L7
In the latest image the environment variables does not work.
The solution is to create a configuration file for the stream.
My working compose file is:
version: '3.7'
configs:
netdata_stream_master:
file: $PWD/stream-master.conf
netdata_stream_client:
file: $PWD/stream-client.conf
services:
netdata-client:
image: netdata/netdata:v1.21.1
hostname: "{{.Node.Hostname}}"
depends_on:
- netdata-central
configs:
-
mode: 444
source: netdata_stream_client
target: /etc/netdata/stream.conf
security_opt:
- apparmor:unconfined
environment:
- PGID=999
volumes:
- /proc:/host/proc:ro
- /etc/passwd:/host/etc/passwd:ro
- /etc/group:/host/etc/group:ro
- /sys:/host/sys:ro
- /var/run/docker.sock:/var/run/docker.sock
deploy:
mode: global
netdata-central:
image: netdata/netdata:v1.21.1
hostname: control
configs:
-
mode: 444
source: netdata_stream_master
target: /etc/netdata/stream.conf
security_opt:
- apparmor:unconfined
environment:
- PGID=999
ports:
- '19999:19999'
volumes:
- /etc/passwd:/host/etc/passwd:ro
- /etc/group:/host/etc/group:ro
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /var/run/docker.sock:/var/run/docker.sock
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
Related
I have the following nodes with hostnames docker-php-pos-web-1,docker-php-pos-web-2,docker-php-pos-web-3,and docker-php-pos-web-4 in a docker swarm cluster with caddy proxy configured on distributed mode
I want requests with cron anywhere in the url path to run on docker-php-pos-web-4. An example request would be demo.phppointofsale.com/index.php/ecommerce/cron. If "cron" is not in the url, it would route as normal.
I want to avoid having 2 copies of production_php_point_of_sale_app just for this.
I am already routing to docker-php-pos-web-4 from my load balancer for "cron" in request path, BUT since in docker swarm the mesh network can decide on which node actually "runs" it. I always want docker-php-pos-web-4 to run these tasks
Below is my docker-compose.yml file
version: '3.9'
services:
production_php_point_of_sale_app:
logging:
driver: "local"
deploy:
restart_policy:
condition: any
mode: global
labels:
caddy: "http://*.phppointofsale.com, http://*.phppos.com"
caddy.reverse_proxy.trusted_proxies: "private_ranges"
caddy.reverse_proxy: "{{upstreams}}"
image: phppointofsale/production-app
build:
context: "production_php_point_of_sale_app"
restart: always
env_file:
- production_php_point_of_sale_app/.env
- .env
networks:
- app_network
- mail
caddy_server:
image: lucaslorentz/caddy-docker-proxy:ci-alpine
ports:
- 80:80
networks:
- caddy_controller
- app_network
environment:
- CADDY_DOCKER_MODE=server
- CADDY_CONTROLLER_NETWORK=10.200.200.0/24
volumes:
- caddy_data:/data
deploy:
restart_policy:
condition: any
mode: global
labels:
caddy_controlled_server:
caddy_controller:
image: lucaslorentz/caddy-docker-proxy:ci-alpine
networks:
- caddy_controller
- app_network
environment:
- CADDY_DOCKER_MODE=controller
- CADDY_CONTROLLER_NETWORK=10.200.200.0/24
volumes:
- /var/run/docker.sock:/var/run/docker.sock
deploy:
restart_policy:
condition: any
placement:
constraints: [node.role == manager]
networks:
caddy_controller:
driver: overlay
ipam:
driver: default
config:
- subnet: "10.200.200.0/24"
app_network:
driver: overlay
mail:
driver: overlay
volumes:
caddy_data: {}
I am trying to deploy my docker stack using compose file. When I deploy with compose file version 3+, the deploy works perfectly fine. But when I am trying to use the 2.4 version or lower I get this error:
unsupported Compose file version: 2.4
I need to use the 2.4 version, because Version 3 and higher does not support several parameters I need for my containers (such as cpuset and runtime).
My version of docker is 19.03.6 and docker-compose is 1.25.4.
Is there any way to deploy with an older version of compose file on Docker 19.03.6? Am I missing something or is the latest docker version does not support the older compose files anymore?
UPDATE
It turns out that docker 19.03.6 supports only Version 3+ in deploy. So I can't use anything but Version 3+, which does not provide the same flexibility as V2.4 in terms of CPU usage setup. The only solution in this situation (when you need parameters like cpuset and runtime) would be to run containers manually or move to something like Kubernetes.
Here are compose files examples:
Version 3.7 (working)
version: '3.7'
services:
mongo:
image: mongo
volumes:
- ~/ProcessingServerData/mongodb/db:/data/db
- ~/ProcessingServerData/mongodb/configdb:/data/configdb
networks:
- proc-net
mongo-express:
image: mongo-express
depends_on:
- mongo
ports:
- 8081:8081
networks:
- proc-net
visualizer:
image: dockersamples/visualizer:stable
ports:
- 8082:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- proc-net
deploy:
placement:
constraints: [node.role == manager]
networks:
proc-net:
driver: overlay
attachable: true
Version 2.4 (not working)
version: '2.4'
services:
mongo:
image: mongo
volumes:
- type: bind
source: ~/ProcessingServerData/mongodb/db
target: /data/db
- type: bind
source: ~/ProcessingServerData/mongodb/configdb
target: /data/configdb
networks:
- proc-net
deploy:
resources:
cpuset: 0,1
mongo-express:
image: mongo-express
depends_on:
- mongo
ports:
- 8081:8081
networks:
- proc-net
deploy:
resources:
cpuset: 0,1
visualizer:
image: dockersamples/visualizer:stable
ports:
- 8082:8080
volumes:
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
networks:
- proc-net
deploy:
resources:
cpuset: 0,1
placement:
constraints: [node.role == manager]
networks:
proc-net:
driver: overlay
deploy config option is not supported in 2.4 , you need to change the file to this one
version: '2.4'
services:
mongo:
image: mongo
volumes:
- type: bind
source: ~/ProcessingServerData/mongodb/db
target: /data/db
- type: bind
source: ~/ProcessingServerData/mongodb/configdb
target: /data/configdb
networks:
- proc-net
mongo-express:
image: mongo-express
depends_on:
- mongo
ports:
- 8081:8081
networks:
- proc-net
visualizer:
image: dockersamples/visualizer:stable
ports:
- 8082:8080
volumes:
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
networks:
- proc-net
networks:
proc-net:
driver: overlay
Apparently there is no support for an older compose file version for deploy.
https://forums.docker.com/t/cant-deploy-stack-with-compose-file-version-2-4-on-docker-19-03-6/90119
I just completed the docker documentation and created two instances on aws (http://13.127.150.218, http://13.235.134.73). The first one is manager and the second one is the worker. Following is the composed file I used to deploy
version: "3"
services:
web:
# replace username/repo:tag with your name and image details
image: username/repo:tag
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "80:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
ports:
- "6379:6379"
volumes:
- "/home/docker/data:/data"
deploy:
placement:
constraints: [node.role == manager]
command: redis-server --appendonly yes
networks:
- webnet
networks:
webnet:
Here the redis service has the constraint that restricts it to run only on manager node. Now my question is how the web service on worker instance is supposed to use the redis service.
You need to use the hostname parameter in all container, so you can use this value to access services from worker or to access from worker the services on manager.
version: "3"
services:
web:
# replace username/repo:tag with your name and image details
image: username/repo:tag
hostname: "web"
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "80:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
hostname: "visualizer"
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
hostname: "redis"
ports:
- "6379:6379"
volumes:
- "/home/docker/data:/data"
deploy:
placement:
constraints: [node.role == manager]
command: redis-server --appendonly yes
networks:
- webnet
networks:
webnet:
In addictional if you use the portainer instead of visualizer you can control you SWARM stack with more options:
https://hub.docker.com/r/portainer/portainer
BR,
Carlos
Consider the stack file as per the below example -
Regardless of where it is placed manager|worker all the services in the stack file being on the same network can use the embedded DNS functionality which helps to resolve each service by the service name defined.
In this case the service web makes use of service redis by its service name.
Here is an example of the ping command able to resolve the service web from within the container associated with the redis service -
Read more about the Swarm Native Service Discovery to understand this.
I've just used the example from this documentation(https://media.readthedocs.org/pdf/portainer/1.19.1/portainer.pdf)
to configure the portainer's agent but keep receiving the error "Unable to proxy the request via the Docker socket". In this case the stack is displayed as "down".
Portainer works just fine without the agent and mounting docker.sock.
This is the docker-compose fragment
agent:
image: portainer/agent
environment:
AGENT_CLUSTER_ADDR: tasks.agent
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- agent_network
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
portainer:
image: portainer/portainer
command: -H tcp://tasks.agent:9001 --tlsskipverify
ports:
- target: 9000
published: 9000
protocol: tcp
mode: ingress
volumes:
- portainer_data:/data
networks:
- agent_network
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
simplified swarm:
manager1 node
- consul-agent
worker1 node
- consul-client1
- web-app:80
- web-network:9000
database1 node
- consul-client2
- redis:6379
- mongo:27017
The web-app and web-network services can connect to redis and mongo through their service names correctly, e.g redis.createClient('6379', 'redis') and mongoose.connect('mongodb://mongo').
However, container web-app cannot connect to web-network, I'm trying to make a request like so:
request('http://web-network:9000')
But get the error:
errorno: ECONNREFUSED
address: 10.0.1.9
port: 9000
Request to web-network using a private IP does work:
request('http://11.22.33.44:9000')
What am I missing? Why can they connect to redis and mongo but not between each container? When moving redis/mongo to the same node as web-app, it will still work, so I don't think the issue comes because the services cannot talk to a service on the same server node.
Can we make docker network use private IP instead of the pre-configured subnet?
docker stack deploy file
version: '3'
services:
web-app:
image: private-repo/private-image
networks:
- swarm-network
ports:
- "80:8080"
deploy:
placement:
constraints:
- node.role==worker
web-network:
image: private-repo/private-image2
networks:
- swarm-network
ports:
- "9000:8080"
deploy:
placement:
constraints:
- node.role==worker
redis:
image: redis:latest
networks:
- swarm-network
ports:
- "6739:6739"
deploy:
placement:
constraints:
- engine.labels.purpose==database
mongo:
image: mongo:latest
networks:
- swarm-network
ports:
- "27017:27017"
deploy:
placement:
constraints:
- engine.labels.purpose==database
networks:
swarm-network:
driver: overlay
docker stack deploy app -c docker-compose.yml