How can I connect my Kubernetes pod to a remote Jaeger? - docker

Question

I am trying to connect my pod from Kubernetes (k8s) cluster to a remote Jaeger server. I've tested and it can work well if both of them are on the same machine. However, when I run my app on k8s, my app can not connect to Jaeger despite I were using physical IP.
First, I've tried this:
containers:
- name: api
env:
- name: OTEL__AGENT_HOST
value: <my-physical-ip>
- name: OTEL__AGENT_PORT
value: "6831"
After read the docs from the internet, I add the Jaeger agent to my deployments as a sidecar container like this.
containers:
- name: api
env:
- name: OTEL__AGENT_HOST
value: "localhost"
- name: OTEL__AGENT_PORT
value: "6831"
- image: jaegertracing/jaeger-agent
name: jaeger-agent
ports:
- containerPort: 5775
protocol: UDP
- containerPort: 6831
protocol: UDP
- containerPort: 6832
protocol: UDP
- containerPort: 5778
protocol: TCP
args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"]
It seems work very well on both containers. But on the collector of Jaeger, I received a log like this:
{"level":"warn","ts":1641987200.2678068,"caller":"channelz/logging.go:62","msg":"[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.
HandleStreams failed to receive the preface from client: read tcp 172.20.0.4:14250-><the-ip-of-machine-my-pods-are-deployed>:32852: i/o timeout\"","system":"grpc","grpc_log":true}

Answer 1

I exposed port 14267 on Jaeger collector on remote machine, then change args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"] to args: ["--reporter.grpc.host-port=<my-physical-ip>:14267"] and it works.

Answer 2

Have you tried using jaeger operator? https://github.com/jaegertracing/jaeger-operator
This is how you will install it :
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.31.0/jaeger-operator.yaml -n observability
then you can create Jaeger instance that will up jaeger components like collector, agent, query . You can define storage too .. like elastic search for e.g.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod-es
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: https://search-test-g7fbo7pzghdquvvgxty2pc6lqu.us-east-2.es.amazonaws.com
index-prefix: jaeger-span
username: test
password: xxxeee
Then in your application's deployment yaml file you will need to configure agent as a side car (or u can use agent as deamonset) so that request can be forwarded to the collector ..
More details here: https://www.jaegertracing.io/docs/1.31/operator/#deployment-strategies