I'm monitoring Java apps with Opentelemetry and exported data to Elastic APM. This integration works well, however we are missing some critical information about metrics.
We want to collect information about the host system and jvm metrics.
Openetelemetry collector is running as sidecar in k8s and its conf is below:
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: app-sidecar
spec:
mode: sidecar
config: |
receivers:
otlp:
protocols:
http:
grpc:
exporters:
logging:
otlp:
endpoint: http://endpoint:8200
headers:
Authorization: Bearer token
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [logging, otlp]
metrics:
receivers: [otlp]
exporters: [logging, otlp]
logs:
receivers: [otlp]
exporters: [logging, otlp]
Start your Java app with java agent opentelemetry-javaagent.jar (OTEL java autoinstrumentation). Configure it to export metrics (it provides by default JVM metrics), for example OTEL_METRICS_EXPORTER=otlp, OTEL_EXPORTER_OTLP_ENDPOINT=<your side car otel collector otlp grpc endpoint>" - check doc for right doc/syntax.
Related
I have a collector running in a docker container with this config file:
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
logging:
loglevel: debug
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging]
logs:
receivers: [otlp]
processors: [batch]
exporters: [logging]
and using this command:
docker run -p 4317:4317 -v /path/to/yaml/file:/etc/otel-collector-config.yaml otel/opentelemetry-collector:latest --config=/etc/otel-collector-config.yaml
and a console app in F# re-written using the same configuration as the C# example here:
https://opentelemetry.io/docs/instrumentation/net/getting-started/#console-application
open System
open System.Diagnostics
open OpenTelemetry.Exporter
open OpenTelemetry
open OpenTelemetry.Trace
open OpenTelemetry.Resources
open Trigger.Reports.Core
[<EntryPoint>]
let main args =
let collectorEndpoint : string = ""
let serviceName : string = "my-service"
let tracerProvider : TracerProvider =
Sdk.CreateTracerProviderBuilder()
.AddOtlpExporter( fun opt ->
opt.Endpoint <- Uri "http://localhost:4317"
)
.AddSource(serviceName)
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(serviceName))
.AddConsoleExporter()
.Build()
let myActivitySource = new ActivitySource(serviceName)
use activity = myActivitySource.StartActivity("SayHello")
activity.SetTag("foo", 1)
0
The traces go to the console exporter but not to the Docker container, from the what I can see I've set both the app and collector up correctly.
Any help would be much appreciated.
I am trying to connect my pod from Kubernetes (k8s) cluster to a remote Jaeger server. I've tested and it can work well if both of them are on the same machine. However, when I run my app on k8s, my app can not connect to Jaeger despite I were using physical IP.
First, I've tried this:
containers:
- name: api
env:
- name: OTEL__AGENT_HOST
value: <my-physical-ip>
- name: OTEL__AGENT_PORT
value: "6831"
After read the docs from the internet, I add the Jaeger agent to my deployments as a sidecar container like this.
containers:
- name: api
env:
- name: OTEL__AGENT_HOST
value: "localhost"
- name: OTEL__AGENT_PORT
value: "6831"
- image: jaegertracing/jaeger-agent
name: jaeger-agent
ports:
- containerPort: 5775
protocol: UDP
- containerPort: 6831
protocol: UDP
- containerPort: 6832
protocol: UDP
- containerPort: 5778
protocol: TCP
args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"]
It seems work very well on both containers. But on the collector of Jaeger, I received a log like this:
{"level":"warn","ts":1641987200.2678068,"caller":"channelz/logging.go:62","msg":"[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.
HandleStreams failed to receive the preface from client: read tcp 172.20.0.4:14250-><the-ip-of-machine-my-pods-are-deployed>:32852: i/o timeout\"","system":"grpc","grpc_log":true}
I exposed port 14267 on Jaeger collector on remote machine, then change args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"] to args: ["--reporter.grpc.host-port=<my-physical-ip>:14267"] and it works.
Have you tried using jaeger operator? https://github.com/jaegertracing/jaeger-operator
This is how you will install it :
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.31.0/jaeger-operator.yaml -n observability
then you can create Jaeger instance that will up jaeger components like collector, agent, query . You can define storage too .. like elastic search for e.g.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod-es
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: https://search-test-g7fbo7pzghdquvvgxty2pc6lqu.us-east-2.es.amazonaws.com
index-prefix: jaeger-span
username: test
password: xxxeee
Then in your application's deployment yaml file you will need to configure agent as a side car (or u can use agent as deamonset) so that request can be forwarded to the collector ..
More details here: https://www.jaegertracing.io/docs/1.31/operator/#deployment-strategies
I am new to Kubernetes. I want to create Service Monitor in Prometheus Operator. I've installed Prometheus Operator and Grafana. I have the running pods as shown below:
The documentation of Prometheus Operator provides code below to create Service Monitor.
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
team: frontend
resources:
requests:
memory: 400Mi
enableAdminAPI: false
I know this is a yaml file, but I am confused how to run this file? In other words, where should I put this piece of code to? I am learning to create Prometheus monitors. Can I get some help?
Thanks!
You can deploy it like any other manifest in Kubernetes cluster, for example by running kubectl apply -f servicemonitor.yaml.
You can check if it was deployed by simply running kubectl get prometheus:
$ kubectl get prometheus
NAME VERSION REPLICAS AGE
prometheus 5s
When you enabled RBAC authorization there is another yaml that you should use:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
labels:
prometheus: prometheus
spec:
replicas: 2
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
team: frontend
alerting:
alertmanagers:
- namespace: default
name: alertmanager
port: web
Installed Spring Cloud Dataflow on Kubernetes (running on DockerDesktop).
Configured Grafana and Prometheus as the per the install guide https://dataflow.spring.io/docs/installation/kubernetes/kubectl/
Created and deployed a simple Stream with time (source) and log (sink) from starter apps .
On selecting Stream dashboard icon in UI, navigates to grafana dashboard but DON'T see the stream and related metrics.
Am I missing any configuration here?
Don't see any action in Prometheus proxy log since it started
scdf-server config map
kind: ConfigMap
apiVersion: v1
metadata:
name: scdf-server
namespace: default
selfLink: /api/v1/namespaces/default/configmaps/scdf-server
uid: ce23d5a3-1cb9-4580-ba1a-bf51b09850dc
resourceVersion: '53607'
creationTimestamp: '2020-04-29T01:28:36Z'
labels:
app: scdf-server
data:
application.yaml: |-
spring:
cloud:
dataflow:
applicationProperties:
stream:
management:
metrics:
export:
prometheus:
enabled: true
rsocket:
enabled: true
host: prometheus-proxy
port: 7001
task:
management:
metrics:
export:
prometheus:
enabled: true
rsocket:
enabled: true
host: prometheus-proxy
port: 7001
grafana-info:
url: 'http://localhost:3000'
task:
platform:
kubernetes:
accounts:
default:
limits:
memory: 1024Mi
datasource:
url: jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT}/mysql
username: root
password: ${mysql-root-password}
driverClassName: org.mariadb.jdbc.Driver
testOnBorrow: true
validationQuery: "SELECT 1"
[Following fixed the Issue]
I updated the stream definition set below in Applications.Properties it started working fine.
management.metrics.export.prometheus.rsocket.host=prometheus-proxy
This metrics collection flow diagram from https://github.com/spring-cloud/spring-cloud-dataflow-samples/tree/master/monitoring-samples helped to spot the issue quick. Thanks
I am new to Prometheus and relatively new to kubernetes so bear with me, please. I am trying to test Prometheus out and have tried two different approaches.
Run Prometheus as a docker container outside of kubernetes. To accomplish this I have created this Dockerfile:
FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/
and this yaml file:
global:
scrape_interval: 15s
external_labels:
monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'kubernetes-apiservers'
scheme: http
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: endpoints
api_server: localhost:443
When I run this I get:
Failed to list *v1.Pod: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
Failed to list *v1.Service: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
Failed to list *v1.Endpoints: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
on a loop. Prometheus will load when I go to localhost:9090 but there is no data.
I thought deploying Prometheus as a Kubernetes deployment may help, so I made this yaml and deployed it.
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: prometheus-monitor
spec:
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
containers:
- name: prometheus-monitor
image: prom/prometheus
# args:
# - '-config.file=/etc/prometheus/prometheus.yaml'
imagePullPolicy: IfNotPresent
ports:
- name: webui
containerPort: 9090
The deployment was successful, but if I go to localhost:9090 I get 'ERR_SOCKET_NOT_CONNECTED'. (my port is forwarded)
Can anyone tell me the advantage of in vs out of Kubernetes and how to fix at least one of these issues?
Also, my config file is suppressed because it was giving an error, and I will look into that once I am able to get Prometheus loaded.
Kubernetes does not map the port outside it's cluster when you deploy your container.
You also have to create a service (can be inside the same file) to make it available from your workstation (append this to your prometheus yaml):
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-web
labels:
app: prometheus
spec:
type: NodePort
ports:
- port: 9090
protocol: TCP
targetPort: 9090
nodePort: 30090
name: webui
selector:
app: prometheus
NodePort opens the given port on all nodes you have. You should be able to see the frontend with http://localhost:30090/
Per default, kubernetes allow ports 30000 to 32767 for NodePort type (https://kubernetes.io/docs/concepts/services-networking/service/#nodeport).
Please consider reading the documentation in general for more information on services in kubernetes: https://kubernetes.io/docs/concepts/services-networking/service/
So 2 different issues. On:
You are trying to connect to localhost:443 where Prometheus is running and it's expecting to talk to a Kubernetes API server. Apparently, nothing is listening on localhost:443. Are you doing port forwarding to your kube-apiserver?
In this case you need to expose your deployment port. With something like:
kubectl expose deployment prmetheus-web --type=LoadBalancer # or
kubectl expose deployment prmetheus-web --type=NodePort
depending on how you want to expose your service. NodePort exposes it in service that maps to a port on your Kubernetes nodes (IPAddress:Port) and LoadBalancer exposes the deployment using an external load balancer that may vary depending on what cloud you are using (AWS, GCP, OpenStack, Azure, etc). More about exposing your Deployments or DaemonSets or StatefulSets here. More about services here
Hope it helps.