I am setting up prometheus + grafana with cadvisor and node_exporter. I need to gather metrics for 2+ servers. But lets say I am gonna do it for 2 not. I am using docker compose.
On Master server I have this configs:
[root#172.31.115.207 ~]# cat prometheus.yml
scrape_configs:
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets:
- prometheus:9090
- pushgateway:9091
- job_name: docker
scrape_interval: 5s
static_configs:
- targets:
- 172.31.115.207:9323
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:8080','172.31.115.207:8080']
- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:9100','172.31.115.207:9100']
cat prometheus.yml
- 172.31.115.207:9323
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:8080','172.31.115.207:8080']
- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:9100','172.31.115.207:9100']
[root#pivanko1c ~]# cat docker-compose.yml
version: '3'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- 9090:9090
command:
- --config.file=/etc/prometheus/prometheus.yml
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
depends_on:
- cadvisor
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
pushgateway:
image: prom/pushgateway
container_name: pushgateway
ports:
- 9091:9091
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- 9100:9100
grafana:
image: grafana/grafana
container_name: grafana
ports:
- 3000:3000
environment:
- GF_SECURITY_ADMIN_PASSWORD=Uniform189
depends_on:
- prometheus
- cadvisor
And here is my docker-compose.yml; on worker server
[root#172.31.112.84 ~]# cat docker-compose.yml
version: '3'
services:
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- 9100:9100
When I have setup only master everything looks nice in grafana. But when I start containers on worker node everything goes wrong in grafana. I only see number of containers. Everything else is N/A.
Please could you advise where is the problem in my configs? I want to have nice view in grafana where I can select node1 or node2 and see all the metrics.
Thank you
Related
I'm having this issue with docker compose running cadvisor and prometheus.
DNS resolution is working for cadvisor
When internet is allowed for cadvisor, prometheus is able to connect to agent using public-ip:8081.
version: '3.7' networks: monitoring:
driver: bridge volumes: prometheus_data: {} services: prometheus:
image: prom/prometheus:latest
user: "1000"
environment:
- PUID=1000
- PGID=1000
container_name: prometheus
restart: unless-stopped
volumes:
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
ports:
- "9090:9090"
networks:
- monitoring
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "127.0.0.1:8081:8080"
# network_mode: "host"
networks:
- monitoring
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- redis
deploy:
mode: global
networks:
- monitoring
```
Is there any allowed-ip list in the cadvisor or a issue with iptables.
yaml
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:8081"]
Tried to using private ip address for cadvisor container, connection refused
Tried using container alias name, connection refused
Published cadvisor contianer to internet, connection accepted to public-ip:8081
I don't want to publish cadvisor over internet.
[RESOLVED]
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:**8080**"]
I thought the mapped port on the left side was used when containers connect to each other. 8081:8080, switched to 8080 and it worked thanks to #anemyte
First of all I tried this solution didn't work for me.
I need to log some custom metrics using Prometheus.
docker-compose.yml
version: "3"
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
hostname: my_service
ports:
- 9090:9090
depends_on:
- my_service
my-service:
build: .
ports:
- 8080:8080
grafana:
image: grafana/grafana:latest
container_name: grafana
hostname: grafana
ports:
- 3000:3000
depends_on:
- prometheus
prometheus.yml
global:
scrape_interval: 5s
scrape_timeout: 10s
external_labels:
monitor: 'my-project'
rule_files:
scrape_configs:
- job_name: myapp
scrape_interval: 10s
static_configs:
- targets:
- my_service:8080
I tried external ip as well, but i can't see my metrics in prometheus UI. Also, the target page is showing localhost:9090 is up.
What could be the problem? Can anyone correct the docker compose and prometheus file?
Thanks
So I found it. I have to set my scrape configs with the container name. something like this
scrape_configs:
- job_name: my-service
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
- 'my-service:8080'
Once you fix your Prometheus volumes to your data, you will see your service is up and running at http://localhost:9090/targets
i should see tree targets in my prometheus dashboard, one from prometheus itself, which works, and one from my self created node.js application called chat-api, and one from cadvisor. for cadvisor i get following error, when i run docker-compose up:
cadvisor | W0419 22:12:08.195849 1 sysinfo.go:203] Nodes topology is not available, providing CPU topology
cadvisor | W0419 22:12:08.196364 1 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
cadvisor | E0419 22:12:08.200398 1 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory
i changed the parameters in my docker-compose file but it dont change anything. im a beginner in docker.
docker-compose.yml:
version : '3.7'
services:
chat-api:
container_name: chat-api
build:
context: .
dockerfile: ./Dockerfile
ports:
- '4000:4000'
networks:
- cchat
restart: 'on-failure'
userdb:
image: mongo:latest
container_name: mongodb
volumes:
- userdb:/data/db
networks:
- cchat
cadvisor:
image: gcr.io/cadvisor/cadvisor
container_name: cadvisor
privileged: true
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker:/var/lib/docker:ro
devices:
- /dev/kmsg:/dev/kmsg
depends_on:
- chat-api
networks:
- cchat
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- prometheus-data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- '9090:9090'
depends_on:
- chat-api
networks:
- cchat
volumes:
userdb:
prometheus-data:
networks:
cchat:
prometheus.yml:
global:
scrape-interval: 5s
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
- job_name: 'chat-api'
static_configs:
- targets: ['chat-api:4000']
Dockerfile:
FROM node:alpine
WORKDIR .
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 4000
CMD ["node", "server.js"]
chat-api is a node application with express
my folder structure:
structure
i am new to prometheus , cadvisor and docker-compose. i made a docker-compose file including my own created application named chat, with a mongo container. those work fine. now i want to monitor my containers with prometheus and cadvisor. im getting following errors:
cadvisor | W0419 11:41:00.576916 1 sysinfo.go:203] Nodes topology is not available, providing CPU topology
cadvisor | W0419 11:41:00.577437 1 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
cadvisor | E0419 11:41:00.582000 1 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory
and
prometheus | ts=2022-04-19T11:54:19.051Z caller=main.go:438 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 2: field scrape-interval not found in type config.plain"
i tryed to change the config parameter from my docker-compose into, but it dont changed the error:
command:
- '--config.file=./prometheus/prometheus.yml'
docker-compose.yml:
version : '3.7'
services:
chat-api:
container_name: chat-api
build:
context: .
dockerfile: ./Dockerfile
ports:
- '4000:4000'
networks:
- cchat
restart: 'on-failure'
userdb:
image: mongo:latest
container_name: mongodb
volumes:
- userdb:/data/db
networks:
- cchat
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- '9080:9080'
networks:
- cloudchat
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker:/var/lib/docker:ro
devices:
- /dev/kmsg:/dev/kmsg
depends_on:
- chat-api
networks:
- cchat
volumes:
userdb:
networks:
cchat:
prometheus.yml:
global:
scrape-interval: 2s
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
project structure:
picture of project structure
I guess it's quite late but you can try mounting /etc/machine-id:/etc/machine-id:ro.
Running in privileged mode could help too. This is my configuration which is working without problems:
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.47.0
container_name: cadvisor
restart: unless-stopped
privileged: true
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
Some important note, don't use latest it seems it's not the latest version (source: https://github.com/google/cadvisor/issues/3066).
I'm trying to get a Prometheus container to scrape metrics from cAdvisor.
This is my prometheus.yml:
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "docker"
static_configs:
- targets: ['localhost:9323']
- job_name: "cadvisor"
scrape_interval: 5s
static_configs:
- targets: ['localhost:7070']
labels:
alias: "cadvisor"
And my docker-compose.yml:
version: "3.5"
services:
app:
container_name: app
build: "./app"
restart: always
volumes:
- ./app:/app
depends_on:
- db
links:
- db
ports:
- 8080:8080
db:
container_name: db
image: postgres
restart: always
volumes:
- ./data/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_DB=articlesdb
- POSTGRES_USER=dbuser
- POSTGRES_PASSWORD=dbpassword
ports:
- "5432:5432"
prometheus:
container_name: prometheus
image: prom/prometheus:latest
#build:
# context: ./prometheus
# dockerfile: Dockerfile
ports:
- 9090:9090
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
depends_on:
- cadvisor
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 7070:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- db
In the targets section Prometheus says that the connection to the cAdvisor container and Docker daemon is refused, while Prometheus' to itself works.
I can reach cAdvisor at localhost:7070/metrics with either cURL or on my browser. What could be the issue, and how can I fix it?
In your docker-compose file you named the cAdvisor service ‘cadvisor’ so in the docker network it can be accessed via the DNS name cadvisor. Change your prometheus.yml static_config like this to scrape the service:
- job_name: "cadvisor"
scrape_interval: 5s
static_configs:
- targets: ['cadvisor:7070']
labels:
alias: "cadvisor"