Relabeling in Prometheus - monitoring

Question

Setup
Prometheus node exporter is registered as a service with consul agent with various tags. Example service definition provided to consul agent:
{
"service":{
"id": "server-stats",
"name": "server-stats",
"tags": [
"a=1_meow",
"b=2_woof",
"c=3_moo",
"monkey"
],
"port": 9100,
"checks": [
{
"name": "Process #1",
"script": "/path/to/healthcheck/script.sh",
"interval": "5s"
}
]
}
}
Prometheus is set to look for this server-stats service and use the configuration (host address and port) provided by Consul to scrape stats from servers. The above tags are available as a comma separated list in __meta_consul_tags that can be used for relabeling.
Prometheus relabeling configuration:
relabel_configs:
- source_labels: [__meta_consul_tags]
separator: ','
#regex: '(.+)=(.+)'
regex: '([a-z_]+)=([a-z_]+|\d+)'
target_label: ${1}
replacement: ${2}
Issue
I am trying to expose tags to Prometheus so that we can get stats and graphs based on labels. Keeping the above service configuration in mind, I would like each metric to have following labels in addition to whatever Prometheus does internally:
a=1_meow, b=2_woof, c=3_moo and ignore monkey because it is just a string. I can remove monkey from my list of tags if there is a solution that requires = to be there. The relabel configuration written above is not leading to exposing any tag at all and seems to be getting ignored. Running Prometheus with log level set to debug is also not yielding anything.
Relevant docs
https://prometheus.io/docs/operating/configuration/#%3Crelabel_config%3E
https://www.robustperception.io/extracting-full-labels-from-consul-tags/

Answer 1

Incorrect understanding
I think there was a mistake in my understanding of how labeling in prometheus works. My incorrect understanding was:
before applying regex, string would be first split on separator (otherwise what is its purpose?),
each substring has regex evaluated against it,
if match groups are declared and found, they will be available as indexed values available to use in target_label and replacement fields.
if regex does not match, then that substring will be ignored.
because regex is expected to be applied to each substring after the split, it will lead to multiple labels from multiple substrings.
Correct understanding
However, from brian-brazil's post linked in his answer and Prometheus's documentation, it seems the following happening:
All __meta tags are combined into one long separator separated line.
regex is applied on that line only once.
If regex matches and includes groups, they are indexed beginning from 1 and available for use in target_label and replacement.
separator seems to be getting ignored in this section even if you mention it.
Config from corrected understanding
From this idea and following from example in the question, I was able to make the following config that works
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([a-z0-9_]+),.+'
target_label: 'a'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,b=([a-z0-9_]+),.+'
target_label: 'b'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,c=([a-z0-9_]+),.+'
target_label: 'c'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,d=([a-z0-9_]+),.+'
target_label: 'd'
replacement: ${1}
Caveats
I believe both approaches (the approach brian-brazil wrote in his blogpost, and what I am using above) have caveats - we either need to know all the labels we want beforehand, or have a set number of them. This means if a developer wants to associate different, or more labels with his/her service, s/he would need to work with ops as general flow will not be able to handle it. I think it is a minor caveat that should be addressed.

Answer 2

https://www.robustperception.io/extracting-full-labels-from-consul-tags/ shows how to do this, in particular the last example.

Answer 3

The following relabeling rule can be used for extracting a particular tag from __meta_consul_tags and placing it to destination_label:
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,tag=([^,]+),.*'
target_label: destination_label
There is no need to specify the replacement option because it defaults to $1.
If multiple tags must be extracted from __meta_consul_tags into multiple labels, then just repeat the relabeling rule per each needed tag. For example, the following relabeling rules extract a tag to a label and b tag to b label:
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([^,]+),.*'
target_label: a
- source_labels: [__meta_consul_tags]
regex: '.*,b=([^,]+),.*'
target_label: b
More information about relabeling rules in Prometheus is available in this article.
Update (2022-12-19): VictoriaMetrics and vmagent (these are Prometheus-like monitoring systems and scrapers I work on) expose __meta_consul_tag_<tagname> and __meta_consul_tagpresent_<tagname> labels for discovered targets starting from v1.85.2. This allows copying all the tags defined at Consul service into the scrape target labels with a single relabeling rule:
- action: labelmap
regex: __meta_consul_tag_(.+)
Try playing with this relabeling rule at Prometheus relabel debugger.