Prometheus create label from metric label - monitoring

Question

We are running the node-exporter in containers. To quickly identify on which host each node-exporter is running, I created a metric that looks like this: host{host="$HOSTNAME",node="$CONTAINER_ID"} 1
I'm looking for a way to extract the hostname in host= and create a label for each node-exporter instance as a hostname label. I tried numerous configurations and none seem to work. Current prometheus config looks like this:
scrape_configs:
- job_name: 'node'
scrape_interval: 10s
scrape_timeout: 5s
metrics_path: /metrics
scheme: http
dns_sd_configs:
- names:
- tasks.master-nodeexporter
refresh_interval: 30s
type: A
port: 9100
relabel_configs:
- source_labels: ['host']
regex: '"(.*)".*'
target_label: 'hostname'
replacement: '$1'

Answer 1

This is not possible, as target relabelling happens before the scrape.
What you want to do here is use service discovery to have the right hostname in the first place, which is not possible with dns_sd_configs. You might look at something like Consul and https://www.robustperception.io/controlling-the-instance-label/

Answer 2

If someone comes across this:
Create this as docker-entrypoint.sh and make it executable.
#!/bin/sh -e
# Must be executable by others
NODE_NAME=$(cat /etc/nodename) echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom
set -- /bin/node_exporter "$#"
exec "$#"
Than create a Dockerfile like this
FROM prom/node-exporter:latest
ENV NODE_ID=none
USER root
COPY conf /etc/node-exporter/
ENTRYPOINT [ "/etc/node-exporter/docker-entrypoint.sh" ]
CMD [ "/bin/node_exporter" ]
Than build it and you will always get the Hostname as a node_meta metric

Answer 3

This answer explains how to export node name via node_name label at node_meta metric. Then it is possible to add node_name label to any metric exposed by node_exporter with group_left() modifier during querying. For example, the following PromQL query adds node_name label from node_meta metric to node_memory_Active_bytes metric:
node_memory_Active_bytes
* on(job,instance) group_left(node_name)
node_meta
See more details about group_left() modifier in these docs and this article.