Prometheus scrape /metric with custom header - monitoring

Question

I have an application that will be monitored by Prometheus,
but the application need the custom header key like :
x-auth-token: <customrandomtoken>
What should I do with prometheus.yml?

Answer 1

Prometheus itself does not have a way to define custom headers in order to reach an exporter. The idea of adding the feature was discussed in this GitHub issue. Tl;dr: if you need a custom header, inject it with a forward proxy (I posted an example in another answer).
The prometheus-blackbox-exporter tag suggest that the question is about the exporter that makes probes, which is a separate thing and it does have a way to set headers. Only, it does not scrape metrics, it makes them.
Blackbox exporter has it's own configuration file and it consists of modules. A module is a set of parameters, defining how to do the probe and what result to expect. Here is an example of a module that looks for 200-299 response code and uses X-Auth-Token header:
modules:
http_2xx_with_header:
prober: http
http:
headers:
X-Auth-Token: skdjfh98732hjf22exampletoken
More examples can be found here and the list of configuration options - here.
When you made the blackbox exporter to load the new configuration, you need to adjust Prometheus configuration as well:
scrape_configs:
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx_with_header] # <- Here goes the name of the new module
static_configs:
- targets:
- http://prometheus.io
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115

Answer 2

Prometheus doesn't support specifying custom HTTP headers, which must be sent with each scrape request to scrape target :( However, it supports specifying Authorization header via authorization and basic_auth options at scrape_config section.
For example, the following config instructs Prometheus to send Authorization: My-Auth my-super-secret header with each scrape request to http://localhost:8428/metrics:
scrape_configs:
- job_name: foo
authorization:
type: "My-Auth"
credentials: "my-super-secret"
static_configs:
- targets: ["localhost:8428"]
P.S. If you still need sending custom http headers with each request to remote target, then take a look at vmagent - monitoring agent, which understands Prometheus scrape configs and can scrape Prometheus targets. This is a project I work on. It provides additional features on top of standard scrape_configs from Prometheus, including headers option - see these docs. The headers option allows specifying arbitrary number of additional http headers, which need to be sent to scrape target. For example, the following config instructs sending x-auth-token: <customrandomtoken> http header with each request to http://foobar:1234:
scrape_configs:
- job_name: foo
headers:
- "x-auth-token: <customrandomtoken>"
static_configs:
- targets: ["foobar:1234"]