My SNMP exporter is hosted somewhere using kubernetes. I can access, and run it through URL like this and get metrics for a specified target: https://some.kube.server.name/api/snmp-exporter/snmp?target=AFACG1
My list of targets is in targets.json file using file_sd_configs in prometheus.yml file to dynamically load targets for prometheus.
My prometheus.yml file looks like, as below:
scrape_configs:
- job_name: 'snmp'
scrape_interval: 120s
file_sd_configs:
- files :
- /etc/prometheus/targets.json
metrics_path: /snmp
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: https://some.kube.server.name/api/snmp-exporter/ # The SNMP exporter's real hostname:port.
And my targets.json file look like, as below:
[
{
"labels": {
"job": "snmp"
},
"targets": [
"AFACG1",
"AFACG3",
"AFACG5",
"AFACG7",
"AFACG8",
"AFACG9"
]
}
]
However, when I run prometheus, I get error \"https://some.kube.server.name/api/snmp-exporter\" is not a valid hostname"
What are the modifications that I need to implement in prometheus.yml in order to get metric for targets in targets.json?
After reading, I figured out an answer to my question which is working out well. Here, what I came up with my modified scrape_configs:
scrape_configs:
- job_name: 'snmp'
scheme: https
scrape_interval: 120s
tls_config:
insecure_skip_verify: true
file_sd_configs:
- files :
- /etc/prometheus/targets.json
metrics_path: /api/snmp-exporter/snmp
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: some.kube.server.name
I hope this helps other people who would face similar problem.
Related
One of the targets in static_configs in my prometheus.yml config file is secured with basic authentication. As a result, an error of description "Connection refused" is always displayed against that target in the Prometheus Targets' page.
I have researched how to setup prometheus to provide the security credentials when trying to scrape that particular target but couldn't find any solution. What I found was how to set it up on the scrape_config section in the docs. This won't work for me because I have other targets that are not protected with basic_auth.
Please help me out with this challenge.
Here is part of my .yml config as regards my challenge.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
scrape_timeout: 5s
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:5000']
labels:
service: 'Auth'
- targets: ['localhost:5090']
labels:
service: 'Approval'
- targets: ['localhost:6211']
labels:
service: 'Credit Assessment'
- targets: ['localhost:6090']
labels:
service: 'Sweep'
- targets: ['localhost:6500']
labels:
I would like to add more details to the #PatientZro answer.
In my case, I need to create another job (as specified), but basic_auth needs to be at the same level of indentation as job_name. See example here.
As well, my basic_auth cases require a path as they are not displayed at the root of my domain.
Here is an example with an API endpoint specified:
- job_name: 'myapp_health_checks'
scrape_interval: 5m
scrape_timeout: 30s
static_configs:
- targets: ['mywebsite.org']
metrics_path: "/api/health"
basic_auth:
username: 'email#username.me'
password: 'cfgqvzjbhnwcomplicatedpasswordwjnqmd'
Best,
Create another job for the one that needs auth.
So just under what you've posted, do another
- job_name: 'prometheus-basic_auth'
scrape_interval: 5s
scrape_timeout: 5s
static_configs:
- targets: ['localhost:5000']
labels:
service: 'Auth'
basic_auth:
username: foo
password: bar
I've a docker swarm configured with nodeA as manager and nodeB as worker and have Promethues installed as docker container on nodeA with prometheus.yml file as -
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets:
- 'localhost:9090'
- job_name: 'node resources'
dns_sd_configs:
- names: ['tasks.node']
type: 'A'
port: 9100
params:
collect[]:
- cpu
- meminfo
- diskstats
- netdev
- netstat
- job_name: 'node storage'
scrape_interval: 1m
dns_sd_configs:
- names: ['tasks.node']
type: 'A'
port: 9100
params:
collect[]:
- filefd
- filesystem
- xfs
- job_name: 'cadvisor'
dns_sd_configs:
- names: ['tasks.cadvisor']
type: 'A'
port: 8080
In Prometheus, if I execute query - container_cpu_usage_seconds_total it gives results like -
✔container_cpu_usage_seconds_total{cpu="cpu15",id="/user.slice",instance="10.0.1.220:8080",job="cadvisor"}
✔container_cpu_usage_seconds_total{cpu="cpu15",id="/user.slice",instance="10.0.1.219:8080",job="cadvisor"}
I can see instance having values 10.0.1.219:8080 and 10.0.1.220:8080 but these are not ip addresses of machines in swarm.
How can I differentiate which instance is for which machine? Is there something which I should configure?
I am trying prometheus to scrape uwsgi container on port 7070
I have following scrape job in prometheus.yaml
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
Also have annotations for uwsgi container :
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "7070"
If I curl http://<host_ip>:7070/metrics from any container, it return json format metrics.
But when I do curl http://<host_ip>:7070/metrics | promtool check metrics.
I get error while linting: text format parsing error in line 1: invalid metric name
I think this is because uwsgi container expose metrics in json format and prometheus doesn't understand json.
What I need to do to make it scrapeable by prometheus ?
Manage to do this with https://github.com/timonwong/uwsgi_exporter as a sidecar container.
- name: uwsgi
image: .....
- name: uwsgi-exporter
image: timonwong/uwsgi-exporter:latest
imagePullPolicy: Always
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- name: uwsgi-exp-port
containerPort: 9117
protocol: TCP
args:
- --stats.uri=http://localhost:7070/metrics
https://www.robustperception.io/writing-json-exporters-in-python
This should be your answer. So you need to write your JSON exporter
I'm trying to connect to get the endpoint metrics via a prometheus docker image. Below is my yml file. However I'm getting the error Get http://localhost:8080/assessments/metrics: dial tcp 127.0.0.1:8080: connect: connection refused from prometheus. It runs if I do it from the browser though. How can I map the port so that docker recognises it.
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
# - "first.rules"
# - "second.rules"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'assessments'
metrics_path: /assessments/metrics
static_configs:
- targets: ['localhost:8080']
~
I was able to fix this by modifying my yml with docker.for.mac.localhost:8080. This made it realise that it had to look for port 8080 in mac
I have a Docker Swarm with a Prometheus container and 1-n containers for a specific microservice.
The microservice-container can be reached by a url. I suppose the requests to this url is kind of load-balanced (of course...).
Currently I have spawned two microservice-container. Querying the metrics now seems to toggle between the two containers. Example: Number of total requests: 10, 13, 10, 13, 10, 13,...
This is my Prometheus configuration. What do I have to do? I do not want to adjust the Prometheus config each time I kill or start a microservice-container.
scrape_configs:
- job_name: 'myjobname'
metrics_path: '/prometheus'
scrape_interval: 15s
static_configs:
- targets: ['the-service-url:8080']
labels:
application: myapplication
UPDATE 1
I changed my configuration as follows which seems to work. This configuration uses a dns lookup inside of the Docker Swarm and finds all instances running the specified service.
scrape_configs:
- job_name: 'myjobname'
metrics_path: '/prometheus'
scrape_interval: 15s
dns_sd_configs:
- names: ['tasks.myServiceName']
type: A
port: 8080
The question here is: Does this configuration recognize that a Docker instance is stopped and another one is started?
UPDATE 2
There is a parameter for what I am asking for:
scrape_configs:
- job_name: 'myjobname'
metrics_path: '/prometheus'
scrape_interval: 15s
dns_sd_configs:
- names: ['tasks.myServiceName']
type: A
port: 8080
# The time after which the provided names are refreshed
[ refresh_interval: <duration> | default = 30s ]
That should do the trick.
So the answer is very simple:
There are multiple, documented ways to scrape.
I am using the dns-lookup-way:
scrape_configs:
- job_name: 'myjobname'
metrics_path: '/prometheus'
scrape_interval: 15s
dns_sd_configs:
- names ['tasks.myServiceName']
type: A
port: 8080
refresh_interval: 15s