Configure prometheus to collect custom metrics from dockerized nodejs pod - docker

Question

I have set up prom-client (unofficial client library for prometheus) to collect custom metrics what I need.
I have prometheus server deployed from helm following this eks setup guide. Now I am trying to edit default configmap to collect my app metrics as well, but getting error
parsing YAML file /etc/config/prometheus.yml: yaml: unmarshal errors:\n line 22: field cluster_ip not found in type kubernetes.plain\n line 25: cannot unmarshal !!strdefaultinto []string
This is what I have done as per docs
prometheus.yaml configmap file
apiVersion: v1
data:
alerting_rules.yml: |
{}
alerts: |
{}
prometheus.yml: |
global:
evaluation_interval: 1m
scrape_interval: 1m
scrape_timeout: 10s
rule_files:
- /etc/config/recording_rules.yml
- /etc/config/alerting_rules.yml
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
...DEFAULT CONFIGS...
- job_name: my_metrics
scrape_interval: 5m
scrape_timeout: 10s
honor_labels: true
metrics_path: /api/metrics
kubernetes_sd_configs:
- role: service
cluster_ip: 10.100.200.92
namespaces:
names:
default
recording_rules.yml: |
{}
rules: |
{}
kind: ConfigMap
metadata:
creationTimestamp: "2020-06-08T09:26:38Z"
labels:
app: prometheus
chart: prometheus-11.3.0
component: server
heritage: Helm
release: prometheus
name: prometheus-server
namespace: prometheus
uid: 8fadb17a-f5c5-4f9d-a931-fa1f77684847
Here clusterIP is the IP assigned for my service to expose deployment.
My deployment.yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 2
strategy:
type: RollingUpdate
selector:
matchLabels:
name: myapp
template:
metadata:
labels:
name: myapp
spec:
containers:
- image: IMAGE_URL:BUILD_NUMBER
name: myapp
resources:
limits:
cpu: "1000m"
memory: "2400Mi"
requests:
cpu: "500m"
memory: "2000Mi"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000
name: myapp
My service.yaml file which is exposing deployment
apiVersion: v1
kind: Service
metadata:
name: myapp
spec:
selector:
deploy: staging
name: myapp
type: ClusterIP
ports:
- port: 80
targetPort: 5000
protocol: TCP
Is there some different/efficient way to target my app for metrics collection, please let me know. Thanks

Answer 1

This is what I am using to enable prometheus scraping inside the cluster.
In the scrape config, I have this snippet:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- action: labeldrop
regex: '(kubernetes_pod|app_kubernetes_io_instance|app_kubernetes_io_name|instance)'
This is taken directly from the default values for the prometheus helm chart: https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L1452
What that does, is instruct prometheus to scrape every pod that has the annotation:
prometheus.io/scrape: "true"
set. With these annotations on the pod you can then configure the port and path of the scrape:
prometheus.io/path: "/metrics"
prometheus.io/port: "9090"
So, you would need to modify your deployment.yaml to specify these annotations as well:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 2
strategy:
type: RollingUpdate
selector:
matchLabels:
name: myapp
template:
metadata:
labels:
name: myapp
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "<enter port of pod to scrape>"
prometheus.io/path: "<enter path to scrape>"
spec:
containers:
- image: IMAGE_URL:BUILD_NUMBER
...