Custom metrics is not showing in prometheus web ui so does in grafana - docker

First of all I tried this solution didn't work for me.
I need to log some custom metrics using Prometheus.
docker-compose.yml
version: "3"
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
hostname: my_service
ports:
- 9090:9090
depends_on:
- my_service
my-service:
build: .
ports:
- 8080:8080
grafana:
image: grafana/grafana:latest
container_name: grafana
hostname: grafana
ports:
- 3000:3000
depends_on:
- prometheus
prometheus.yml
global:
scrape_interval: 5s
scrape_timeout: 10s
external_labels:
monitor: 'my-project'
rule_files:
scrape_configs:
- job_name: myapp
scrape_interval: 10s
static_configs:
- targets:
- my_service:8080
I tried external ip as well, but i can't see my metrics in prometheus UI. Also, the target page is showing localhost:9090 is up.
What could be the problem? Can anyone correct the docker compose and prometheus file?
Thanks

So I found it. I have to set my scrape configs with the container name. something like this
scrape_configs:
- job_name: my-service
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
- 'my-service:8080'
Once you fix your Prometheus volumes to your data, you will see your service is up and running at http://localhost:9090/targets

Related

Docker cadvisor connection refused

I'm having this issue with docker compose running cadvisor and prometheus.
DNS resolution is working for cadvisor
When internet is allowed for cadvisor, prometheus is able to connect to agent using public-ip:8081.
version: '3.7' networks: monitoring:
driver: bridge volumes: prometheus_data: {} services: prometheus:
image: prom/prometheus:latest
user: "1000"
environment:
- PUID=1000
- PGID=1000
container_name: prometheus
restart: unless-stopped
volumes:
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
ports:
- "9090:9090"
networks:
- monitoring
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "127.0.0.1:8081:8080"
# network_mode: "host"
networks:
- monitoring
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- redis
deploy:
mode: global
networks:
- monitoring
```
Is there any allowed-ip list in the cadvisor or a issue with iptables.
yaml
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:8081"]
Tried to using private ip address for cadvisor container, connection refused
Tried using container alias name, connection refused
Published cadvisor contianer to internet, connection accepted to public-ip:8081
I don't want to publish cadvisor over internet.
[RESOLVED]
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:**8080**"]
I thought the mapped port on the left side was used when containers connect to each other. 8081:8080, switched to 8080 and it worked thanks to #anemyte

Prometheus keeps showing up on 9090 even if docker container not running

Following a tutorial. I have the following docker compose:
version: "3.9"
services:
grafana:
image: grafana/grafana
ports:
- 3000:3000
prometheus:
image: prom/prometheus
ports:
- 9090:9090
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
postgres:
image: postgres:12
ports:
- 5432:5432
volumes:
- ./backup:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: postgrespassword
POSTGRES_DB: shop
postgres-exporter:
image: prometheuscommunity/postgres-exporter
ports:
- 9187:9187
environment:
DATA_SOURCE_NAME: "postgresql://postgres:postgrespassword#postgres:5432/shop?sslmode=disable"
links:
- postgres
- prometheus
And this as my prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
- job_name: postgres-exporter
static_configs:
- targets: ["postgres-exporter:9187"]
I realized I needed to make some changes, so brought down the containers. But noticed I could still access:
http://xxx.xx.xx.xxx:9090/config
In checking the containers, it seemed to restart Prometheus. I disabled it again, checked on the browser, and saw it got restarted again. Additionally, any changes I make to prometheus.yml aren't reflected in the config.
I checked running services, there is no instance of prometheus running outside of docker.
Disabling the docker service brings it down, but as soon as I re-enable it, it appears again, always with the old config.
any ideas? (have tried killing the port as well, just respawns)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
34c66a78b75e prom/prometheus:latest "/bin/prometheus --c…" 2 hours ago Up 2 hours 9090/tcp my-prometheus.1.iht6uk1okzlb3gc9y9jdaf2fg

Prometheus + Grafana on multiple nodes using docker compose

I am setting up prometheus + grafana with cadvisor and node_exporter. I need to gather metrics for 2+ servers. But lets say I am gonna do it for 2 not. I am using docker compose.
On Master server I have this configs:
[root#172.31.115.207 ~]# cat prometheus.yml
scrape_configs:
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets:
- prometheus:9090
- pushgateway:9091
- job_name: docker
scrape_interval: 5s
static_configs:
- targets:
- 172.31.115.207:9323
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:8080','172.31.115.207:8080']
- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:9100','172.31.115.207:9100']
cat prometheus.yml
- 172.31.115.207:9323
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:8080','172.31.115.207:8080']
- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets: ['172.31.112.84:9100','172.31.115.207:9100']
[root#pivanko1c ~]# cat docker-compose.yml
version: '3'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- 9090:9090
command:
- --config.file=/etc/prometheus/prometheus.yml
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
depends_on:
- cadvisor
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
pushgateway:
image: prom/pushgateway
container_name: pushgateway
ports:
- 9091:9091
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- 9100:9100
grafana:
image: grafana/grafana
container_name: grafana
ports:
- 3000:3000
environment:
- GF_SECURITY_ADMIN_PASSWORD=Uniform189
depends_on:
- prometheus
- cadvisor
And here is my docker-compose.yml; on worker server
[root#172.31.112.84 ~]# cat docker-compose.yml
version: '3'
services:
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
ports:
- 9100:9100
When I have setup only master everything looks nice in grafana. But when I start containers on worker node everything goes wrong in grafana. I only see number of containers. Everything else is N/A.
Please could you advise where is the problem in my configs? I want to have nice view in grafana where I can select node1 or node2 and see all the metrics.
Thank you

Cannot access host machine from docker container

Using docker version 18.09.2. Using docker on windows 10.
Setting up a prometheus and grafana stack to monitor metrics on a service running on my localhost. Here's my docker compose file.
version: '3.4'
networks:
monitor-net:
driver: bridge
dockernet:
external: true
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:v2.7.1
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
restart: unless-stopped
expose:
- 9090
networks:
- monitor-net
- dockernet
extra_hosts:
- "localhost1:10.0.75.1"
labels:
org.label-schema.group: "monitoring"
grafana:
image: grafana/grafana:5.4.3
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources:/etc/grafana/datasources
- ./grafana/dashboards:/etc/grafana/dashboards
- ./grafana/setup.sh:/setup.sh
entrypoint: /setup.sh
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
- dockernet
labels:
org.label-schema.group: "monitoring"
caddy:
image: stefanprodan/caddy
container_name: caddy
ports:
- "3000:3000"
- "9090:9090"
- "9093:9093"
- "9091:9091"
volumes:
- ./caddy/:/etc/caddy/
environment:
- ADMIN_USER=${ADMIN_USER:-admin}
- ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
restart: unless-stopped
networks:
- monitor-net
- dockernet
labels:
org.label-schema.group: "monitoring"
Here is my prometheus.yml file.
global:
scrape_interval: 15s
evaluation_interval: 15s
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'docker-host-alpha'
# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
- "alert.rules"
# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9090']
- job_name: 'myapp'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:32771']
- job_name: 'myapp1'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.internal:51626']
- job_name: 'myapp2'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.internal.localhost:51626']
- job_name: 'myapp3'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.localhost:51626']
- job_name: 'myapp4'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:51626']
- job_name: 'myapp5'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['host.docker.internal:51626']
- job_name: 'myapp6'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['host.docker.internal.localhost:51626']
- job_name: 'myapp7'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:51626']
- job_name: 'myapp8'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['127.0.0.1:51626']
- job_name: 'myapp9'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['localhost:51626']
- job_name: 'myapp10'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['10.0.75.1:51626']
- job_name: 'myapp12'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['localhost1:51626']
From what I understand host.docker.internal should reference my host IP and give me access to my local app but it didn't. So then I looked up my docker NAT IP address with ipconfig (the 10.0.75.1 address) and that didn't work either.
Then I tried the network binding of localhost1 to 10.0.75.1. I tried setting up a bridge network called dockernet and connect that way and it didn't work. When I launch my app in a docker container I can get to it through "docker.for.win.localhost:32771" but this container can't access my remote database so that's why I need it to run local.
Prometheus gives the following responses for some of the respective addresses:
Endpoint: Error
http://docker.for.win.localhost:32771/metrics: UP
http://host.docker.internal:51626/metrics: server returned HTTP status 400 Bad Request
http://docker.for.win.localhost:51626/metrics: server returned HTTP status 400 Bad Request
http://host.docker.internal.localhost:51626/metrics: Get http://host.docker.internal.localhost:51626/metrics: dial tcp: lookup host.docker.internal.localhost on 127.0.0.11:53: no such host
http://docker.for.win.host.internal.localhost:51626/metrics: Get http://docker.for.win.host.internal.localhost:51626/metrics: dial tcp: lookup docker.for.win.host.internal.localhost on 127.0.0.11:53: no such host
I've tried everything and am out of ideas. Can anyone shed some light?
I have the similar problem. I locally run my own application on IIS Express on port 52562, and prometheus inside container show that http://docker.for.win.localhost:52562/metrics return 400 BAD Request.
Problem was that IIS Express listen only for localhost, so I edit bindings in my applicationhost.config from
<binding protocol="http" bindingInformation="*:52562:localhost" />
to
<binding protocol="http" bindingInformation="*:52562:" />
and restart IIS Express.
This fixed the problem.
for my works this
version: "3"
networks:
sandbox:
driver: bridge
services:
prometheus:
restart: always
image: prom/prometheus:v2.3.2
volumes: ["./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"]
ports: ["9090:9090"]
extra_hosts: ["host.docker.internal:172.17.0.1"] # from Gateway bridge and added /etc/hosts
networks: ["sandbox"]
grafana:
....
You might also check if your windows 10 firewall is blocking the connection.
To disable the firewall completely:
netsh advfirewall set allprofiles state off
To allow a connection on a specific port:
New-NetFirewallRule -Protocol TCP -LocalPort 44369 -Direction Inbound -Action Allow -DisplayName "Allow network TCP on port 44369"

Prometheus - Target Connection refused

I'm trying to get a Prometheus container to scrape metrics from cAdvisor.
This is my prometheus.yml:
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "docker"
static_configs:
- targets: ['localhost:9323']
- job_name: "cadvisor"
scrape_interval: 5s
static_configs:
- targets: ['localhost:7070']
labels:
alias: "cadvisor"
And my docker-compose.yml:
version: "3.5"
services:
app:
container_name: app
build: "./app"
restart: always
volumes:
- ./app:/app
depends_on:
- db
links:
- db
ports:
- 8080:8080
db:
container_name: db
image: postgres
restart: always
volumes:
- ./data/postgres:/var/lib/postgresql/data
environment:
- POSTGRES_DB=articlesdb
- POSTGRES_USER=dbuser
- POSTGRES_PASSWORD=dbpassword
ports:
- "5432:5432"
prometheus:
container_name: prometheus
image: prom/prometheus:latest
#build:
# context: ./prometheus
# dockerfile: Dockerfile
ports:
- 9090:9090
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
depends_on:
- cadvisor
cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
ports:
- 7070:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- db
In the targets section Prometheus says that the connection to the cAdvisor container and Docker daemon is refused, while Prometheus' to itself works.
I can reach cAdvisor at localhost:7070/metrics with either cURL or on my browser. What could be the issue, and how can I fix it?
In your docker-compose file you named the cAdvisor service ‘cadvisor’ so in the docker network it can be accessed via the DNS name cadvisor. Change your prometheus.yml static_config like this to scrape the service:
- job_name: "cadvisor"
scrape_interval: 5s
static_configs:
- targets: ['cadvisor:7070']
labels:
alias: "cadvisor"

Resources