We are trying to Autoscale the Pods using HorizontalPodScaler, we are trying to scale based on Prometheus, we have set up Prometheus instance but unable to write metrics to Prometheus using Kafka exporter, is there any clear approach for doing this?
Related
Trying to determine if cAdvisor + Prometheus is the OTHER option for monitoring openshift containers. Or if there is another combo that I can use natively from Prometheus.
cAdvisor is essentially built into K8S deployments so it's as native as you can get really. If you want to use additional software there are other ways to collect data with agents, but cAdvisor is quite well understood and efficient for doing this type of data collection. Prometheus also scrapes other K8S APIs aside from cAdvisor.
I have a unique type of Kubernetes cluster that cannot install the Kubernetes Datadog agent. I would like to collect the logs of individual docker containers in my Kubernetes pods similar to how the Docker agent works.
I am currently collecting docker logs from Kubernetes and then using a script with the Datadog custom log forwarder to upload them to Datadog. I was curious if there is a better way to achieve this serverless collection of docker logs from Kubernetes clusters in datadog? The ideal situation I want is to plug my kubeconfig somewhere and then let Datadog take care of the rest without deploying anything onto my Kubernetes cluster.
Is there an option for that outside of creating a custom script?
A better way would be to use a sidecar container with a logging agent, it won't increase the load on the API server.
Reference: https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent
Datadog agent looks like doesn't support /suggest running as a sidecar (https://github.com/DataDog/datadog-agent/issues/2203#issuecomment-416180642)
I suggest looking at using other logging agent and pointing the backend to datadog.
Some options are:
fluentd: https://blog.powerupcloud.com/kubernetes-pod-management-using-fluentd-as-a-sidecar-container-and-prestop-lifecycle-hook-part-iv-428b5f4f7fc7
fluentd-bit: https://github.com/leahnp/fluentbit-sidecar
filebeat: https://www.elastic.co/beats/filebeat
Datadog supports them
https://docs.datadoghq.com/integrations/fluentd/
https://docs.datadoghq.com/integrations/filebeat/
Currently I am using an external server to connect into Kubernetes cluster using a KUBECONFIG file. I have a custom docker image build on this external server(not on master/worker nodes).
I need a solution to use this custom image to create a deployment in K8S cluster by using k8s client libraries only (or simply using yaml kubectl create steps)
Is there a way for it ?
I have multiple java application running inside the container and all of them is managed by kubernetes.
I am using prometheus to monitor cotainer level metrics i.e cpu, mem, etc.
Now I want to do applciation level monitoring using jmx_exporter. but with every deploy container IP keep changing.
can I some how use kuberentes service-ip(cluster-ip) which don't change.
I cannot just directly put kube-service ip as it load-balance it among containers. and every time i will get only one contaienr metrics insted of all.
or is there a way to dynamically discover cotainer with service_name or replication_controller name in prometheus.
Yes, you can scrape pods in kubernetes.
You can find an example of how to do that here
we are planning to use cadvisor for collecting cgroup data from docker host. we have set up using collectd and grafana to chart monitoring of other app metrics.
anyone know plugins for cadvisor plugin for collectd ? as per my info collectd can not pull data of cgroup docker containers.
cadvisor has influxdb support, and grafana can hook up to influxdb for metrics visualization. but as we do not have influxdb in place in current landscape we are exploring quick approach for docker container metrics monitoring.
Thanks in advance
cAdvisor plugin for collectd would be pretty simple. Can you file an issue on github.com/google/cadvisor and we can help you get one.
Alternatively, you can always hit the cAdvisor rest endpoint for the whole machine or a particular container to pull the data into graphite and push it from a helper process. eg.
/api/v1.3/containers/
In any case, file a feature request and we can help you out with the setup.
You can use fluentd to collect data from cadvisor using fluent-plugin-cadvisor.
Maybe it is not the best plugin, but it's very easy to extend and add it as you own plugin to fluentd.