I'm having this issue with docker compose running cadvisor and prometheus.
DNS resolution is working for cadvisor
When internet is allowed for cadvisor, prometheus is able to connect to agent using public-ip:8081.
version: '3.7' networks: monitoring:
driver: bridge volumes: prometheus_data: {} services: prometheus:
image: prom/prometheus:latest
user: "1000"
environment:
- PUID=1000
- PGID=1000
container_name: prometheus
restart: unless-stopped
volumes:
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
ports:
- "9090:9090"
networks:
- monitoring
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "127.0.0.1:8081:8080"
# network_mode: "host"
networks:
- monitoring
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- redis
deploy:
mode: global
networks:
- monitoring
```
Is there any allowed-ip list in the cadvisor or a issue with iptables.
yaml
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:8081"]
Tried to using private ip address for cadvisor container, connection refused
Tried using container alias name, connection refused
Published cadvisor contianer to internet, connection accepted to public-ip:8081
I don't want to publish cadvisor over internet.
[RESOLVED]
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:**8080**"]
I thought the mapped port on the left side was used when containers connect to each other. 8081:8080, switched to 8080 and it worked thanks to #anemyte
Related
First of all I tried this solution didn't work for me.
I need to log some custom metrics using Prometheus.
docker-compose.yml
version: "3"
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
hostname: my_service
ports:
- 9090:9090
depends_on:
- my_service
my-service:
build: .
ports:
- 8080:8080
grafana:
image: grafana/grafana:latest
container_name: grafana
hostname: grafana
ports:
- 3000:3000
depends_on:
- prometheus
prometheus.yml
global:
scrape_interval: 5s
scrape_timeout: 10s
external_labels:
monitor: 'my-project'
rule_files:
scrape_configs:
- job_name: myapp
scrape_interval: 10s
static_configs:
- targets:
- my_service:8080
I tried external ip as well, but i can't see my metrics in prometheus UI. Also, the target page is showing localhost:9090 is up.
What could be the problem? Can anyone correct the docker compose and prometheus file?
Thanks
So I found it. I have to set my scrape configs with the container name. something like this
scrape_configs:
- job_name: my-service
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
- 'my-service:8080'
Once you fix your Prometheus volumes to your data, you will see your service is up and running at http://localhost:9090/targets
On my Windows 10 Pro I have a complete Docker Swarm environment. Part of the Docker Swarm stack are Prometheus and cAdvisor. Step by step I will build the monitoring tools and then deploy the monitoring to a Cloud solution.
In the Docker Swarm stack I can run Prometheus and Cadvisor, but Prometheus cannot connect to cAdvisor.
I get the message:
Get http://cadvisor:8090/metrics: dial tcp 10.0.0.50:8090: connect:
connection refused
How can I get Prometheus access cadvisor?
In my browser I can perform a 'localhost:8090/metrics' and get all metrics. So, the cAdvisor runs for sure.
I have one stack file that creates the network (devhome_default). In my second stack I refer to this network.
UPDATE: one way to solve this is to use the IP-address: $ ipconfig
Using that address in my prometheus.config works fine. But this makes the target hard-wired and not maintainable.
The stack / dockercompose file is:
version: '3'
services:
cadvisor:
image: google/cadvisor
networks:
- geosolutionsnet
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /:/rootfs
- /var/run:/var/run
- /sys:/sys
- /var/lib/docker/:/var/lib/docker
ports:
- 8090:8080
deploy:
mode: global
resources:
limits:
cpus: '0.10'
memory: 128M
reservations:
cpus: '0.10'
memory: 64M
prometheus:
image: prom/prometheus:v2.8.0
ports:
- "9090:9090"
networks:
- geosolutionsnet
volumes:
- //k/data/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
deploy:
mode: replicated
replicas: 1
networks:
geosolutionsnet:
external:
name: devhome_default
The prometheus config file is:
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
#- "alert.rules_nodes"
#- "alert.rules_tasks"
#- "alert.rules_service-groups"
scrape_configs:
- job_name: 'prometheus'
dns_sd_configs:
- names:
- 'tasks.prometheus'
type: 'A'
port: 9090
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8090']
labels:
alias: "cadvisor"
Alternatively, I tried also for cAdvisor:
- job_name: 'cadvisor'
dns_sd_configs:
- names:
- 'tasks.cadvisor'
type: 'A'
port: 8090
And also:
- job_name: 'cadvisor'
static_configs:
- targets: ['localhost:8090']
In different Cloud environments the below 'dns' solution works like charm. Because a 'real' Cloud environment is the target environment for our Docker containers, the standard 'de facto' solution suffices.
So, this works well in Cloud environments:
- job_name: 'cadvisor'
dns_sd_configs:
- names:
- 'tasks.cadvisor'
type: 'A'
port: 8090
I am setting up prometheus + grafana with cadvisor and node_exporter. I need to gather metrics for 2+ servers. But lets say I am gonna do it for 2 not. I am using docker compose.
On Master server I have this configs:
[root#172.31.115.207 ~]# cat prometheus.yml
scrape_configs:
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets:
- prometheus:9090
- pushgateway:9091
- job_name: docker
scrape_interval: 5s
static_configs:
- targets:
- 172.31.115.207:9323
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:8080','172.31.115.207:8080']
- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:9100','172.31.115.207:9100']
cat prometheus.yml
- 172.31.115.207:9323
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:8080','172.31.115.207:8080']
- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:9100','172.31.115.207:9100']
[root#pivanko1c ~]# cat docker-compose.yml
version: '3'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- 9090:9090
command:
- --config.file=/etc/prometheus/prometheus.yml
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
depends_on:
- cadvisor
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
pushgateway:
image: prom/pushgateway
container_name: pushgateway
ports:
- 9091:9091
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- 9100:9100
grafana:
image: grafana/grafana
container_name: grafana
ports:
- 3000:3000
environment:
- GF_SECURITY_ADMIN_PASSWORD=Uniform189
depends_on:
- prometheus
- cadvisor
And here is my docker-compose.yml; on worker server
[root#172.31.112.84 ~]# cat docker-compose.yml
version: '3'
services:
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- 9100:9100
When I have setup only master everything looks nice in grafana. But when I start containers on worker node everything goes wrong in grafana. I only see number of containers. Everything else is N/A.
Please could you advise where is the problem in my configs? I want to have nice view in grafana where I can select node1 or node2 and see all the metrics.
Thank you
Using docker version 18.09.2. Using docker on windows 10.
Setting up a prometheus and grafana stack to monitor metrics on a service running on my localhost. Here's my docker compose file.
version: '3.4'
networks:
monitor-net:
driver: bridge
dockernet:
external: true
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:v2.7.1
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
restart: unless-stopped
expose:
- 9090
networks:
- monitor-net
- dockernet
extra_hosts:
- "localhost1:10.0.75.1"
labels:
org.label-schema.group: "monitoring"
grafana:
image: grafana/grafana:5.4.3
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources:/etc/grafana/datasources
- ./grafana/dashboards:/etc/grafana/dashboards
- ./grafana/setup.sh:/setup.sh
entrypoint: /setup.sh
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
- dockernet
labels:
org.label-schema.group: "monitoring"
caddy:
image: stefanprodan/caddy
container_name: caddy
ports:
- "3000:3000"
- "9090:9090"
- "9093:9093"
- "9091:9091"
volumes:
- ./caddy/:/etc/caddy/
environment:
- ADMIN_USER=${ADMIN_USER:-admin}
- ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
restart: unless-stopped
networks:
- monitor-net
- dockernet
labels:
org.label-schema.group: "monitoring"
Here is my prometheus.yml file.
global:
scrape_interval: 15s
evaluation_interval: 15s
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'docker-host-alpha'
# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
- "alert.rules"
# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9090']
- job_name: 'myapp'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:32771']
- job_name: 'myapp1'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.internal:51626']
- job_name: 'myapp2'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.internal.localhost:51626']
- job_name: 'myapp3'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.localhost:51626']
- job_name: 'myapp4'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:51626']
- job_name: 'myapp5'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['host.docker.internal:51626']
- job_name: 'myapp6'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['host.docker.internal.localhost:51626']
- job_name: 'myapp7'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:51626']
- job_name: 'myapp8'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['127.0.0.1:51626']
- job_name: 'myapp9'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['localhost:51626']
- job_name: 'myapp10'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['10.0.75.1:51626']
- job_name: 'myapp12'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['localhost1:51626']
From what I understand host.docker.internal should reference my host IP and give me access to my local app but it didn't. So then I looked up my docker NAT IP address with ipconfig (the 10.0.75.1 address) and that didn't work either.
Then I tried the network binding of localhost1 to 10.0.75.1. I tried setting up a bridge network called dockernet and connect that way and it didn't work. When I launch my app in a docker container I can get to it through "docker.for.win.localhost:32771" but this container can't access my remote database so that's why I need it to run local.
Prometheus gives the following responses for some of the respective addresses:
Endpoint: Error
http://docker.for.win.localhost:32771/metrics: UP
http://host.docker.internal:51626/metrics: server returned HTTP status 400 Bad Request
http://docker.for.win.localhost:51626/metrics: server returned HTTP status 400 Bad Request
http://host.docker.internal.localhost:51626/metrics: Get http://host.docker.internal.localhost:51626/metrics: dial tcp: lookup host.docker.internal.localhost on 127.0.0.11:53: no such host
http://docker.for.win.host.internal.localhost:51626/metrics: Get http://docker.for.win.host.internal.localhost:51626/metrics: dial tcp: lookup docker.for.win.host.internal.localhost on 127.0.0.11:53: no such host
I've tried everything and am out of ideas. Can anyone shed some light?
I have the similar problem. I locally run my own application on IIS Express on port 52562, and prometheus inside container show that http://docker.for.win.localhost:52562/metrics return 400 BAD Request.
Problem was that IIS Express listen only for localhost, so I edit bindings in my applicationhost.config from
<binding protocol="http" bindingInformation="*:52562:localhost" />
to
<binding protocol="http" bindingInformation="*:52562:" />
and restart IIS Express.
This fixed the problem.
for my works this
version: "3"
networks:
sandbox:
driver: bridge
services:
prometheus:
restart: always
image: prom/prometheus:v2.3.2
volumes: ["./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"]
ports: ["9090:9090"]
extra_hosts: ["host.docker.internal:172.17.0.1"] # from Gateway bridge and added /etc/hosts
networks: ["sandbox"]
grafana:
....
You might also check if your windows 10 firewall is blocking the connection.
To disable the firewall completely:
netsh advfirewall set allprofiles state off
To allow a connection on a specific port:
New-NetFirewallRule -Protocol TCP -LocalPort 44369 -Direction Inbound -Action Allow -DisplayName "Allow network TCP on port 44369"
I'm trying to get a Prometheus container to scrape metrics from cAdvisor.
This is my prometheus.yml:
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "docker"
static_configs:
- targets: ['localhost:9323']
- job_name: "cadvisor"
scrape_interval: 5s
static_configs:
- targets: ['localhost:7070']
labels:
alias: "cadvisor"
And my docker-compose.yml:
version: "3.5"
services:
app:
container_name: app
build: "./app"
restart: always
volumes:
- ./app:/app
depends_on:
- db
links:
- db
ports:
- 8080:8080
db:
container_name: db
image: postgres
restart: always
volumes:
- ./data/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_DB=articlesdb
- POSTGRES_USER=dbuser
- POSTGRES_PASSWORD=dbpassword
ports:
- "5432:5432"
prometheus:
container_name: prometheus
image: prom/prometheus:latest
#build:
# context: ./prometheus
# dockerfile: Dockerfile
ports:
- 9090:9090
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
depends_on:
- cadvisor
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 7070:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- db
In the targets section Prometheus says that the connection to the cAdvisor container and Docker daemon is refused, while Prometheus' to itself works.
I can reach cAdvisor at localhost:7070/metrics with either cURL or on my browser. What could be the issue, and how can I fix it?
In your docker-compose file you named the cAdvisor service ‘cadvisor’ so in the docker network it can be accessed via the DNS name cadvisor. Change your prometheus.yml static_config like this to scrape the service:
- job_name: "cadvisor"
scrape_interval: 5s
static_configs:
- targets: ['cadvisor:7070']
labels:
alias: "cadvisor"