docker-compose.yml: Service is up and running.
version: '3'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- 9090:9090
volumes:
- ./prometheus/conf:/etc/prometheus
# - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./alertmanager/alert_rules.yml:/alertmanager/alert.rules.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
ports:
- 9093:9093
volumes:
- ./alertmanager/:/etc/alertmanager/
command:
- '--config.file=/etc/alertmanager/alertmanager.yml'
- '--storage.path=/alertmanager'
prometheus.yml: Prometheus config file with targets and alerts target sets. The alertmanager target url is working fine.
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
- 10.10.5.14:9093
scheme: http
timeout: 10s
api_version: v1
rule_files:
- "/alertmanager/alert.rules:/alertmanager/alert.rules"
scrape_configs:
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- "google.com"
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115
alert.rules: Standard alert rule.
groups:
- name: example
rules:
# Alert for any instance that is unreachable for >1 minutes.
- alert: InstanceDown
expr: up == 0
for: 1m
alertmanager.yml: Code to send alerts via gmail smtp.
route:
group_by: [Alertname]
# Send all notifications to me.
receiver: email-me
receivers:
- name: email-me
email_configs:
- to: anyemail.com
from: senderlogin#gmail.com
smarthost: smtp.gmail.com:587
auth_username: "senderlogin#gmail.com"
auth_identity: "senderlogin#gmail.com"
auth_password: "password"
Problem: All containers are working but there are no rule in prometheus rules.
And the letters do not come accordingly either. I can’t understand what I missed. Any hints are welcome. Thanks.
enter image description here
enter image description here
Alertmanager is not on localhost from the view of the prometheus container, the networking is namespaced in containers so you need to contact alertmanager using DNS discovery to find the current container IP. With compose, DNS is automatically configured for the service name, in this case alertmanager:
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
Related
I'm having this issue with docker compose running cadvisor and prometheus.
DNS resolution is working for cadvisor
When internet is allowed for cadvisor, prometheus is able to connect to agent using public-ip:8081.
version: '3.7' networks: monitoring:
driver: bridge volumes: prometheus_data: {} services: prometheus:
image: prom/prometheus:latest
user: "1000"
environment:
- PUID=1000
- PGID=1000
container_name: prometheus
restart: unless-stopped
volumes:
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
ports:
- "9090:9090"
networks:
- monitoring
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "127.0.0.1:8081:8080"
# network_mode: "host"
networks:
- monitoring
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- redis
deploy:
mode: global
networks:
- monitoring
```
Is there any allowed-ip list in the cadvisor or a issue with iptables.
yaml
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:8081"]
Tried to using private ip address for cadvisor container, connection refused
Tried using container alias name, connection refused
Published cadvisor contianer to internet, connection accepted to public-ip:8081
I don't want to publish cadvisor over internet.
[RESOLVED]
global:
scrape_interval: 1m
scrape_configs:
- job_name: "prometheus" (Working)
scrape_interval: 1m
static_configs:
- targets: ["localhost:9090"]
- job_name: "node" (Working)
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "cadvisor" (not working)
scrape_interval: 5s
static_configs:
- targets: ["cadvisor:**8080**"]
I thought the mapped port on the left side was used when containers connect to each other. 8081:8080, switched to 8080 and it worked thanks to #anemyte
First of all I tried this solution didn't work for me.
I need to log some custom metrics using Prometheus.
docker-compose.yml
version: "3"
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
hostname: my_service
ports:
- 9090:9090
depends_on:
- my_service
my-service:
build: .
ports:
- 8080:8080
grafana:
image: grafana/grafana:latest
container_name: grafana
hostname: grafana
ports:
- 3000:3000
depends_on:
- prometheus
prometheus.yml
global:
scrape_interval: 5s
scrape_timeout: 10s
external_labels:
monitor: 'my-project'
rule_files:
scrape_configs:
- job_name: myapp
scrape_interval: 10s
static_configs:
- targets:
- my_service:8080
I tried external ip as well, but i can't see my metrics in prometheus UI. Also, the target page is showing localhost:9090 is up.
What could be the problem? Can anyone correct the docker compose and prometheus file?
Thanks
So I found it. I have to set my scrape configs with the container name. something like this
scrape_configs:
- job_name: my-service
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
- 'my-service:8080'
Once you fix your Prometheus volumes to your data, you will see your service is up and running at http://localhost:9090/targets
I have a following compose file namesd docker-compose.yaml:
version: '3.7'
networks:
back-tier:
services:
prometheus:
image: prom/prometheus:v2.36.2
volumes:
- ./prometheus/:/prometheus/
ports:
- 9090:9090
networks:
- back-tier
restart: always
Folder structure looks like this:
[![enter image description here][1]][1]
prometheus.yml looks like this:
# Sample Prometheus config
# This assumes that your Prometheus instance can access this application on localhost:8080
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
scrape_configs:
- job_name: 'spring boot scrape'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:8080']
I've started it with
docker-compoe up -d
I have spring boot application started locally:
when I hit http://localhost:8080/actuator/prometheus
I see following(my custom metric):
....
# HELP greeting_time_seconds Time taken to return greeting
# TYPE greeting_time_seconds summary
greeting_time_seconds{class="com.tutorialworks.demos.springbootwithmetrics.GreetingController",exception="none",method="greeting",quantile="0.5",} 8.8064E-5
greeting_time_seconds{class="com.tutorialworks.demos.springbootwithmetrics.GreetingController",exception="none",method="greeting",quantile="0.9",} 0.022018048
greeting_time_seconds_count{class="com.tutorialworks.demos.springbootwithmetrics.GreetingController",exception="none",method="greeting",} 9.0
greeting_time_seconds_sum{class="com.tutorialworks.demos.springbootwithmetrics.GreetingController",exception="none",method="greeting",} 0.022103
# HELP greeting_time_seconds_max Time taken to return greeting
# TYPE greeting_time_seconds_max gauge
greeting_time_seconds_max{class="com.tutorialworks.demos.springbootwithmetrics.GreetingController",exception="none",method="greeting",} 0.0212423
....
But when I try to access prometheus
http://localhost:9090/
I don't see any greeting_time_seconds metric
So looks like prometheus can't gather my metrics.
I've checked http://localhost:9090/config
And it responds with:
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
alerting:
alertmanagers:
- follow_redirects: true
enable_http2: true
scheme: http
timeout: 10s
api_version: v2
static_configs:
- targets: []
scrape_configs:
- job_name: prometheus
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
enable_http2: true
static_configs:
- targets:
- localhost:9090
So looks like it doesn't see my config from local folder.
How can I fix it ?
UPDATE
I've corrected volume in docker-compose file:
....
prometheus:
image: prom/prometheus:v2.36.2
volumes:
- ./prometheus/:/etc/prometheus/
....
And now http://localhost:9090/
responds with configuration I provided:
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
scrape_configs:
- job_name: spring boot scrape
honor_timestamps: true
scrape_interval: 5s
scrape_timeout: 5s
metrics_path: /actuator/prometheus
scheme: http
follow_redirects: true
enable_http2: true
static_configs:
- targets:
- localhost:8080
but prometheus UI doesn't return my greeting metric and offers only limited list of metrics:
UPDATE#2:
according to advice of DazWilkin I've updated compose file with
version: '3.7'
#networks:
# back-tier:
services:
prometheus:
image: prom/prometheus:v2.36.2
volumes:
- ./prometheus/:/etc/prometheus/
ports:
- 9090:9090
# networks:
# - back-tier
network_mode: "host"
restart: always
And after compose up I see that container is alive but URL localhost:9090 is not accessible now
[1]: https://i.stack.imgur.com/gAtOE.jpg
Finally I was able to find a solution:
In prometheus config I had to provide host.docker.internal instead of localhost
scrape_configs:
- job_name: 'spring boot scrape'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['host.docker.internal:8080']
I am playing with Grafana and Promtail. I have the following setup:
version: "3.3"
networks:
loki:
services:
loki:
image: grafana/loki:k88-c660a7e
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki
promtail:
image: grafana/promtail:k88-c660a7e
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers
- /var/run/docker.sock:/var/run/docker.sock
- ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
command: -config.file=/etc/promtail/promtail-config.yaml
networks:
- loki
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
volumes:
- ./grafana-config.yaml:/etc/grafana/provisioning/datasources/default.yaml
networks:
- loki
mycontainer: # Prints debug output to stdout
build: .
labels:
- "mylabel=true"
networks:
- loki
What I want to achieve:
Have Promtail ignore all other containers except ones that have the mylabel=true
Have Grafana display a label with the container name as label in the log explorer
My current promtail-config.yaml scrape-configs:
scrape_configs:
- job_name: containers
docker_sd_configs:
- host: unix:///var/run/docker.sock
relabel_configs: # For some reason this drops all logs
- source_labels: [__meta_docker_container_label_mylabel]
regex: "true"
action: keep
static_configs:
- targets:
- localhost
labels:
job: containerlogs
__path__: /var/lib/docker/containers/*/*-json.log
pipeline_stages:
- docker:
How do I change it to achieve that?
EDIT: This is probably going to come up - support for docker_sd_configs option is not officially released yet, but I have confirmed with a contributor to the project that it is available and in fact running in their cloud.
Use filters for docker labels selection. Use relabel_configs for setting additional labels based on meta values
server:
http_listen_address: 0.0.0.0
http_listen_port: 9080
positions:
filename: /var/run/promtail/positions.yaml
clients:
- url: https://[YOUR_LOKI_SERVER_NAME]/loki/api/v1/push
tenant_id: [YOUR_TENANT_ID]
scrape_configs:
- job_name: containers
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 15s
filters:
- name: label
values: ["mylabel=true"]
pipeline_stages:
- docker: {}
- static_labels:
job: "promtail"
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
Using docker version 18.09.2. Using docker on windows 10.
Setting up a prometheus and grafana stack to monitor metrics on a service running on my localhost. Here's my docker compose file.
version: '3.4'
networks:
monitor-net:
driver: bridge
dockernet:
external: true
volumes:
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:v2.7.1
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
restart: unless-stopped
expose:
- 9090
networks:
- monitor-net
- dockernet
extra_hosts:
- "localhost1:10.0.75.1"
labels:
org.label-schema.group: "monitoring"
grafana:
image: grafana/grafana:5.4.3
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources:/etc/grafana/datasources
- ./grafana/dashboards:/etc/grafana/dashboards
- ./grafana/setup.sh:/setup.sh
entrypoint: /setup.sh
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
- dockernet
labels:
org.label-schema.group: "monitoring"
caddy:
image: stefanprodan/caddy
container_name: caddy
ports:
- "3000:3000"
- "9090:9090"
- "9093:9093"
- "9091:9091"
volumes:
- ./caddy/:/etc/caddy/
environment:
- ADMIN_USER=${ADMIN_USER:-admin}
- ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
restart: unless-stopped
networks:
- monitor-net
- dockernet
labels:
org.label-schema.group: "monitoring"
Here is my prometheus.yml file.
global:
scrape_interval: 15s
evaluation_interval: 15s
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'docker-host-alpha'
# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
- "alert.rules"
# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9090']
- job_name: 'myapp'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:32771']
- job_name: 'myapp1'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.internal:51626']
- job_name: 'myapp2'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.internal.localhost:51626']
- job_name: 'myapp3'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.host.localhost:51626']
- job_name: 'myapp4'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:51626']
- job_name: 'myapp5'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['host.docker.internal:51626']
- job_name: 'myapp6'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['host.docker.internal.localhost:51626']
- job_name: 'myapp7'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['docker.for.win.localhost:51626']
- job_name: 'myapp8'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['127.0.0.1:51626']
- job_name: 'myapp9'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['localhost:51626']
- job_name: 'myapp10'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['10.0.75.1:51626']
- job_name: 'myapp12'
scrape_interval: 10s
metrics_path: /metrics
static_configs:
- targets: ['localhost1:51626']
From what I understand host.docker.internal should reference my host IP and give me access to my local app but it didn't. So then I looked up my docker NAT IP address with ipconfig (the 10.0.75.1 address) and that didn't work either.
Then I tried the network binding of localhost1 to 10.0.75.1. I tried setting up a bridge network called dockernet and connect that way and it didn't work. When I launch my app in a docker container I can get to it through "docker.for.win.localhost:32771" but this container can't access my remote database so that's why I need it to run local.
Prometheus gives the following responses for some of the respective addresses:
Endpoint: Error
http://docker.for.win.localhost:32771/metrics: UP
http://host.docker.internal:51626/metrics: server returned HTTP status 400 Bad Request
http://docker.for.win.localhost:51626/metrics: server returned HTTP status 400 Bad Request
http://host.docker.internal.localhost:51626/metrics: Get http://host.docker.internal.localhost:51626/metrics: dial tcp: lookup host.docker.internal.localhost on 127.0.0.11:53: no such host
http://docker.for.win.host.internal.localhost:51626/metrics: Get http://docker.for.win.host.internal.localhost:51626/metrics: dial tcp: lookup docker.for.win.host.internal.localhost on 127.0.0.11:53: no such host
I've tried everything and am out of ideas. Can anyone shed some light?
I have the similar problem. I locally run my own application on IIS Express on port 52562, and prometheus inside container show that http://docker.for.win.localhost:52562/metrics return 400 BAD Request.
Problem was that IIS Express listen only for localhost, so I edit bindings in my applicationhost.config from
<binding protocol="http" bindingInformation="*:52562:localhost" />
to
<binding protocol="http" bindingInformation="*:52562:" />
and restart IIS Express.
This fixed the problem.
for my works this
version: "3"
networks:
sandbox:
driver: bridge
services:
prometheus:
restart: always
image: prom/prometheus:v2.3.2
volumes: ["./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"]
ports: ["9090:9090"]
extra_hosts: ["host.docker.internal:172.17.0.1"] # from Gateway bridge and added /etc/hosts
networks: ["sandbox"]
grafana:
....
You might also check if your windows 10 firewall is blocking the connection.
To disable the firewall completely:
netsh advfirewall set allprofiles state off
To allow a connection on a specific port:
New-NetFirewallRule -Protocol TCP -LocalPort 44369 -Direction Inbound -Action Allow -DisplayName "Allow network TCP on port 44369"