Node Exporter Basic Auth (docker-compose) - docker

I want to set up node exporter on my server to be monitored using docker compose but do not want the metrics to be freely available to all.
My current docker-compose.yml file looks like this;
version: '3.8'
networks:
monitoring:
driver: bridge
volumes:
prometheus_data: {}
services:
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- '--path.procfs=/host/proc'
- '--path.rootfs=/rootfs'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
- '--collector.netclass.ignored-devices=^(veth.*)$$'
ports:
- 9100:9100
networks:
- monitoring
labels:
org.label-schema.group: "monitoring"
When I add bottom line to my docker-compose.yml file then I get error message "services.node-exporter Additional property basic_auth_users is not allowed".
basic_auth_users:
prometheus: my_pass
can someone please help me where I am making mistakes or how the whole thing would work.
Ps: I would like to install on the server to be monitored only Node-Exporter since a Prometheus instance is not necessary there.... (Correct me if it is wrong approach)
Best regards

Solution -> docker-compose.yml
version: '3.8'
networks:
monitoring:
driver: bridge
volumes:
prometheus_data: {}
services:
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
- ./prometheus/web.yml:/etc/prometheus/web.yml
command:
- '--path.procfs=/host/proc'
- '--path.rootfs=/rootfs'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
- '--collector.netclass.ignored-devices=^(veth.*)$$'
- '--web.config=/etc/prometheus/web.yml'
ports:
- 9100:9100
networks:
- monitoring
labels:
org.label-schema.group: "monitoring"

Related

ElasticSearch with Docker and Traefic

When I will run elasticsearch with docker and traefic (With SSL encryption).
Than I can't connect to elasticsearch via domain.
When I remove all traefic things in the docker-compose by the elastic search tham it works over the IP and http.
Here is my docker-compose.yml:
`
version: "3.7"
networks:
traefik:
external: true
search:
services:
elasticsearch:
image: elasticsearch:7.16.2
container_name: elasticsearch
environment:
- xpack.security.enabled=false
- discovery.type=single-node
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=XXXXXX
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
labels:
- "traefik.enable=true"
- "traefik.http.routers.elastic_search.rule=Host(se.mydomain.xyz)"
- "traefik.http.routers.elastic_search.entrypoints.websecure"
- "traefik.http.services.elastic_search.loadbalancer.server.port=9200"
ports:
- 9200:9200
- 9300:9300
networks:
- search
kibana:
image: kibana:7.16.2
container_name: kibana
labels:
- "traefik.enable=true"
- "traefik.http.routers.kibana_search.rule=Host(kibana.mydomain.xyz)"
- "traefik.http.routers.kibana_search.entrypoints.websecure"
- "traefik.http.services.kibana_search.loadbalancer.server.port=5601"
#ports:
#- 5601:5601
environment:
ELASTICSEARCH_URL: http://XXX.XXX.XXX.XXX:9200
ELASTICSEARCH_HOSTS: '["http://XXX.XXX.XXX.XXX:9200"]'
networks:
- search
- traefik
volumes:
elasticsearch-data:
driver: local
`
Has anyone a idea for a solution?

Set up Prometheus and Cadvisor with docker-compose

i am new to prometheus , cadvisor and docker-compose. i made a docker-compose file including my own created application named chat, with a mongo container. those work fine. now i want to monitor my containers with prometheus and cadvisor. im getting following errors:
cadvisor | W0419 11:41:00.576916 1 sysinfo.go:203] Nodes topology is not available, providing CPU topology
cadvisor | W0419 11:41:00.577437 1 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
cadvisor | E0419 11:41:00.582000 1 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory
and
prometheus | ts=2022-04-19T11:54:19.051Z caller=main.go:438 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 2: field scrape-interval not found in type config.plain"
i tryed to change the config parameter from my docker-compose into, but it dont changed the error:
command:
- '--config.file=./prometheus/prometheus.yml'
docker-compose.yml:
version : '3.7'
services:
chat-api:
container_name: chat-api
build:
context: .
dockerfile: ./Dockerfile
ports:
- '4000:4000'
networks:
- cchat
restart: 'on-failure'
userdb:
image: mongo:latest
container_name: mongodb
volumes:
- userdb:/data/db
networks:
- cchat
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- '9080:9080'
networks:
- cloudchat
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker:/var/lib/docker:ro
devices:
- /dev/kmsg:/dev/kmsg
depends_on:
- chat-api
networks:
- cchat
volumes:
userdb:
networks:
cchat:
prometheus.yml:
global:
scrape-interval: 2s
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
project structure:
picture of project structure
I guess it's quite late but you can try mounting /etc/machine-id:/etc/machine-id:ro.
Running in privileged mode could help too. This is my configuration which is working without problems:
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.47.0
container_name: cadvisor
restart: unless-stopped
privileged: true
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
Some important note, don't use latest it seems it's not the latest version (source: https://github.com/google/cadvisor/issues/3066).

Why am I unable to route to my API backend with Traefik

I had two container frontend (nginx :80) and backend (nodejs :3000).
I'm trying to redirect all path to my frontend : localhost/* to my frontend
Except one path to my backend API : localhost/v1/* to my backend
I secure my database container (mongodb) by allowing only communication with my backend
Here is my docker-compose.yml (I'm only using this)
version: '3'
services:
traefik:
image: traefik:v2.3
container_name: traefik
command:
- --api.insecure=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --entrypoints.web.address=:80
ports:
- "8080:8080"
- "443:443"
- "80:80"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
frontend:
image: registry.gitlab.com/test/frontend
container_name: frontend
build:
context: ../frontend/.
labels:
- traefik.enable=true
- traefik.http.routers.frontend.rule=PathPrefix(`/`)
- traefik.http.routers.frontend.entrypoints=web
networks:
- traefik-network
backend:
image: registry.gitlab.com/test/backend
container_name: backend
build:
context: ../backend/.
labels:
- traefik.enable=true
- traefik.http.routers.backend.rule=PathPrefix(`/v1`)
- traefik.http.routers.backend.service=backend
- traefik.http.routers.backend.entrypoints=web
- traefik.http.services.backend.loadbalancer.server.port=3000
command: yarn start
environment:
- MONGODB_URL=mongodb://mongodb:27017/backend
depends_on:
- mongodb
volumes:
- ../backend/.:/usr/src/backend
networks:
- traefik-network
- backend-network
mongodb:
image: mongo:4.2.1-bionic
container_name: mongodb
ports:
- 27017:27017
volumes:
- dbdata:/data/db
networks:
- backend-network
volumes:
dbdata:
networks:
backend-network:
traefik-network:
The problem is...
If the frontend (backend and traefik too) is turned on
the paths to localhost/* work (this is what I want),
but the paths to localhost/v1/* don't work (Problem here!).
If the frontend is turned off but traefik and backend is turned on
the paths to localhost/* don't work (of course, that's right),
but the paths to localhost/v1/* work (of course, this is what I want).
I've tried a lot of solutions but nothing seems to work the way I want it to.
What did I misunderstand?
Thanks for helping,
Have a nice day
Try to add the following labels to the backend service
- "traefik.http.routers.backend.rule=Host(`servicex.me`) && Path(`/v1`)"
and frontend
- traefik.http.routers.frontend.rule=Host(`servicex.me`)
you also need to add this line to your /etc/hosts
127.0.0.1 servicex.me
and make sure that you stop and start the services
Complete Example
version: '3'
services:
traefik:
image: traefik:v2.3
container_name: traefik
command:
- --api.insecure=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --entrypoints.web.address=:80
ports:
- "8080:8080"
- "443:443"
- "80:80"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
frontend:
image: registry.gitlab.com/test/frontend
container_name: frontend
build:
context: ../frontend/.
labels:
- traefik.enable=true
- traefik.http.routers.frontend.rule=Host(`servicex.me`)
- traefik.http.routers.frontend.entrypoints=web
- traefik.http.routers.frontend.service=frontend
- traefik.http.services.frontend.loadbalancer.server.port=80
networks:
- traefik-network
backend:
image: registry.gitlab.com/test/backend
container_name: backend
build:
context: ../backend/.
labels:
- traefik.enable=true
- "traefik.http.routers.backend.rule=Host(`servicex.me`) && Path(`/v1`)"
- traefik.http.routers.backend.service=backend
- traefik.http.routers.backend.entrypoints=web
- traefik.http.services.backend.loadbalancer.server.port=3000
command: yarn start
environment:
- MONGODB_URL=mongodb://mongodb:27017/backend
depends_on:
- mongodb
volumes:
- ../backend/.:/usr/src/backend
networks:
- traefik-network
- backend-network
mongodb:
image: mongo:4.2.1-bionic
container_name: mongodb
ports:
- 27017:27017
volumes:
- dbdata:/data/db
networks:
- backend-network
volumes:
dbdata:
networks:
backend-network:
traefik-network:
BTW, why do you need both traefik and nginx (Both are doing the same job), it would be better if you can replace one with another.
I added this label to my containers
traefik.docker.network=traefik-network
It works fine now

multiple docker compose files with traefik (v2.1) and database networks

I would like to build a docker landscape. I use a container with a traefik (v2. 1) image and a mysql container for multiple databases.
traefik/docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
restart: always
command:
- "--log.level=DEBUG"
- "--api=true"
- "--api.dashboard=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=proxy"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--entrypoints.traefik-dashboard.address=:8080"
- "--certificatesresolvers.devnik-resolver.acme.httpchallenge=true"
- "--certificatesresolvers.devnik-resolver.acme.httpchallenge.entrypoint=web"
#- "--certificatesresolvers.devnik-resolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.devnik-resolver.acme.email=####"
- "--certificatesresolvers.devnik-resolver.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- "./letsencrypt:/letsencrypt"
- "./data:/etc/traefik"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
- "proxy"
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik.rule=Host(`devnik.dev`)"
- "traefik.http.routers.traefik.entrypoints=traefik-dashboard"
- "traefik.http.routers.traefik.tls.certresolver=devnik-resolver"
#basic auth
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.middlewares=auth"
- "traefik.http.middlewares.auth.basicauth.usersfile=/etc/traefik/.htpasswd"
#Docker Networks
networks:
proxy:
database/docker-compose.yml
version: "3.3"
services:
#MySQL Service
mysql:
image: mysql:5.7
container_name: mysql
restart: always
ports:
- "3306:3306"
volumes:
#persist data
- ./mysqldata/:/var/lib/mysql/
- ./init:/docker-entrypoint-initdb.d
networks:
- "mysql"
environment:
MYSQL_ROOT_PASSWORD: ####
TZ: Europe/Berlin
#Docker Networks
networks:
mysql:
driver: bridge
For the structure I want to control all projects via multiple docker-compose files. These containers should run on the same network as the traefik container and some with the mysql container.
This also works for the following case (but only sometimes)
dev-releases/docker-compose.yml
version: "3.3"
services:
backend:
image: "registry.gitlab.com/devnik/dev-releases-backend/master:latest"
container_name: "dev-releases-backend"
restart: always
volumes:
#laravel logs
- "./logs/backend:/app/storage/logs"
#cron logs
- "./logs/backend/cron.log:/var/log/cron.log"
labels:
- "traefik.enable=true"
- "traefik.http.routers.dev-releases-backend.rule=Host(`dev-releases.backend.devnik.dev`)"
- "traefik.http.routers.dev-releases-backend.entrypoints=websecure"
- "traefik.http.routers.dev-releases-backend.tls.certresolver=devnik-resolver"
networks:
- proxy
- mysql
environment:
TZ: Europe/Berlin
#Docker Networks
networks:
proxy:
external:
name: "traefik_proxy"
mysql:
external:
name: "database_mysql"
As soon as I restart the containers in dev-releases/ via docker-compose up -d I get the typical error "Gateway timeout" when calling them in the browser.
As soon as I comment the network networks: #- mysql and restart the docker-compose in dev-releases/ it works again.
My guess is that I have not configured the external networks correctly. Is it not possible to use 2 external networks?
I'd like some container have access to the 'mysql' network but it should not be accessible for the whole traefik network.
Let me know if you need more information
EDIT (26.03.2020)
I make it running.
I put all my containers into one network "proxy". It seems mysql also have to be in the proxy network.
So I add following to database/docker-compose.yml
networks:
proxy:
external:
name: "traefik_proxy"
And removed the database_mysql network out of dev-releases/docker-compose.yml
based on the names of the files, your mysql network should be mysql_mysql.
you can verify this by executing
$> docker network ls
You are also missing a couple of labels for your services such as
traefik command line
- '--providers.docker.watch=true'
- '--providers.docker.swarmMode=true'
labels
- traefik.docker.network=proxy
- traefik.http.services.dev-releases-backend.loadbalancer.server.port=yourport
- traefik.http.routers.dev-releases-backend.service=mailcatcher
You can check this for more info

What is wrong in my docker-compose.yml file?

I need a little bit of help to find why my docker-compose.yml doesn't work. The story is that I have a docker-compose.yml who works (and who create Grafana, Prometheus, nodeexporter, cAdvisor and alertManager) then I want to do one without Grafana so I just removed all the Grafana things of the file but it doesn't work.
the one who works :
version: '2'
networks:
monitor-net:
driver: bridge
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '-config.file=/etc/prometheus/prometheus.yml'
- '-storage.local.path=/prometheus'
- '-alertmanager.url=http://alertmanager:9093'
- '-storage.local.memory-chunks=100000'
restart: unless-stopped
expose:
- 9090
ports:
- 9090:9090
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
alertmanager:
image: prom/alertmanager
container_name: alertmanager
volumes:
- ./alertmanager/:/etc/alertmanager/
command:
- '-config.file=/etc/alertmanager/config.yml'
- '-storage.path=/alertmanager'
restart: unless-stopped
expose:
- 9093
ports:
- 9093:9093
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
nodeexporter:
image: prom/node-exporter
container_name: nodeexporter
restart: unless-stopped
expose:
- 9100
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
cadvisor:
image: google/cadvisor:v0.24.1
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
restart: unless-stopped
expose:
- 8080
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
grafana:
image: grafana/grafana
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
env_file:
- user.config
restart: unless-stopped
expose:
- 3000
ports:
- 3000:3000
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
The one who doesn't work:
version: '2'
networks:
monitor-net:
driver: bridge
volumes:
prometheus_data: {}
services:
prometheus:
image: prom/prometheus
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '-config.file=/etc/prometheus/prometheus.yml'
- '-storage.local.path=/prometheus'
- '-alertmanager.url=http://alertmanager:9093'
- '-storage.local.memory-chunks=100000'
restart: unless-stopped
expose:
- 9090
ports:
- 9090:9090
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
alertmanager:
image: prom/alertmanager
container_name: alertmanager
volumes:
- ./alertmanager/:/etc/alertmanager/
command:
- '-config.file=/etc/alertmanager/config.yml'
- '-storage.path=/alertmanager'
restart: unless-stopped
expose:
- 9093
ports:
- 9093:9093
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
nodeexporter:
image: prom/node-exporter
container_name: nodeexporter
restart: unless-stopped
expose:
- 9100
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
cadvisor:
image: google/cadvisor
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
restart: unless-stopped
expose:
- 8080
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
With the second one the container Prometheus can't run and the logs are:
level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75"
level=info msg="Build context (go=go1.7.5, user=root#1a01c5f68840, date=20170220-07:00:00)" source="main.go:76"
level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:248"
level=error msg="Error opening memory series storage: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000009]" source="main.go:182"
The fact that Prometheus at least starts and then errors out means that your compose files is probably correct.
It seems to at least try to load the configuration file /etc/prometheus/prometheus.yml and fails doing so.
In the compose file I see that it adds a host volume which is expected to exist on your host system at location ./prometheus/. Did you also copy this folder and it's contents? If yes, did you verify that the configuration is correct and is expected to work without Grafana? Also it's important what your current directory is when you run docker-compose, it must be where the ./prometheus/ directory is located.

Resources