I have a JVM applicaiton running in kubernetes. When I run kubectl top pod I can see the following
mypod1 12m 6035Mi
mypod2 11m 6129Mi
mypod3 11m 6334Mi
I would like to find out whether that 6GB memory is good or bad. My kubernetes deployment yaml does not specify any resources
Questions
How can I find out the maximum number that it can get to?
How can I find out whether jvm is performing well?
Is there a profiler I can connect to the jvm running in the pods?
Question #1: "How can I find out the maximum number that it can get to?"
A: Without resources configured in the deployment, a pod will have QoS class of BestEffort and can use as much memory as it is available on the node where it is running. See also my answer to this question: How can I tell how much RAM my Kubernetes pod has?
It is always a good practice, IMHO, to at least specify the min (-Xms) and max (-Xmx) JVM heap...
Question #2: "How can I find out whether jvm is performing well?"
A: You can start with enabling JMX and then using it to collect JVM and application metrics. Besides the JMX-to-HTTP bridges like Jolokia and Prometheus JMX Exporter, it is also an option to connect directly over JMX. One way is to:
Expose JMX by configuring these JVM startup arguments:
-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=4444 -Djava.rmi.server.hostname=127.0.0.1
Notice that this affixes the otherwise dynamic RMI port and sets hostname for the RMI server.
Forward local ports to these ports on the pod:
kubectl --namespace=<namespace> port-forward <pod-name> 4444:4444 1099:1099
Start locally a tool that can connect to the JVM in the pod over JMX (jconsole, jvisualvm, jmc...based on what's available to you). The JMX URL would be:
service:jmx:rmi://127.0.0.1:4444/jndi/rmi://127.0.0.1:1099/jmxrmi
Question #3: "Is there a profiler I can connect to the jvm running in the pods?"
A: The short answer is "yes". I have used JProfiler to remotely profile Java apps running on k8s through port forwarding. (I am not affiliated with JProfiler nor am promoting it - it was simply the tool the team I was helping had a license for)
You can use the Vertical Pod Autoscaler to get CPU/Memory limits automatically updated as per the consumption of your container.
To get JVM metrics I would recommend installing some prometheus plugin in your server and scraping the metrics. Then you can see how many objects are alive and generational information to better understand the cause of memory usage and how to control it.
You can connect a profiler by exposing the relevant ports using kubectl proxy
Related
Here in Cassandra deployment instruction, it says:
Caution:
Minikube defaults to 2048MB of memory and 2 CPU. Running Minikube with
the default resource configuration results in insufficient resource
errors during this tutorial. To avoid these errors, start Minikube
with the following settings:
minikube start --memory 5120 --cpus=4
But I don't know what is the equivalent command if I use Kind instead of minikube?
I also like to know if I want to set these settings permanently for my Kind cluster, how should I do that?
kind does not work like minikube, it inherit the resources from the host. If you want to limit the resources then create an vm with specified limit and create kind cluster in it. You can check similar discussion on this thread
I am new to docker and I hope to monitor some QoS metrics of service in a docker container. Does docker provide some API that I can count the HTTP requests to each container with deploy code inside of the containers?
I believe Docker's only metrics API is the /containers/{id}/stats endpoint, and it looks like that only publishes interface-level statistics (generally the total number of bytes that have gone in and out of a container). That interface doesn't provide port-level or HTTP-level metrics.
You might check to see whether the HTTP framework you're using has metric instrumentation built-in. If it does, you can use open-source tools like Prometheus and Grafana to collect and display those metrics.
At a technical level, decoding the HTTP stream is trickier than just forwarding packets along, and you need a more involved proxy setup. If you're using Kubernetes, service meshes like Istio often provide these statistics for you, by running a proxy (Envoy) that handles all traffic; but especially for a small application, instrumenting your own code will be much easier than installing Kubernetes and Istio and trying to deploy there.
I am creating a docker container ( using docker run) in a kubernetes Environment by invoking a rest API.
I have mounted the docker.sock of the host machine and i am building an image and running that image from RESTAPI..
Now i need to connect to this container from some other container which is actually started by Kubectl from deployment.yml file.
But when used kubeclt describe pod (Pod name), my container created using Rest API is not there.. So where is this container running and how can i connect to it from some other container ?
Are you running the container in the same namespace as namespace with deployment.yml? One of the option to check that would be to run -
kubectl get pods --all-namespaces
If you are not able to find the docker container there than I would suggest performing below steps -
docker ps -a {verify running docker status}
Ensuring that while mounting docker.sock there are no permission errors
If there are permission errors, escalate privileges to the appropriate level
To answer the second question, connection between two containers should be possible by referencing cluster DNS in below format -
"<servicename>.<namespacename>.svc.cluster.local"
I would also request you to detail steps, codes and errors(if there are any) for me to better answer the question.
You probably shouldn't be directly accessing the Docker API from anywhere in Kubernetes. Kubernetes will be totally unaware of anything you manually docker run (or equivalent) and as you note normal administrative calls like kubectl get pods won't see it; the CPU and memory used by the pod won't be known about by the node interface and this could cause a node to become over utilized. The Kubernetes network environment is also pretty complicated, and unless you know the details of your specific CNI provider it'll be hard to make your container accessible at all, much less from a pod running on a different node.
A process running in a pod can access the Kubernetes API directly, though. That page notes that all of the official client libraries are aware of the conventions this uses. This means that you should be able to directly create a Job that launches your target pod, and a Service that connects to it, and get the normal Kubernetes features around this. (For example, servicename.namespacename.svc.cluster.local is a valid DNS name that reaches any Pod connected to the Service.)
You should also consider whether you actually need this sort of interface. For many applications, it will work just as well to deploy some sort of message-queue system (e.g., RabbitMQ) and then launch a pool of workers that connects to it. You can control the size of the worker queue using a Deployment. This is easier to develop since it avoids a hard dependency on Kubernetes, and easier to manage since it prevents a flood of dynamic jobs from overwhelming your cluster.
The Kubernetes documentation states it's possible to use Elasticsearch and Kibana for cluster level logging.
Is this possible to do this on the instance of Kubernetes that's shipped with Docker for Windows as per the documentation? I'm not interested in third party Kubernetes manifests or Helm charts that mimic this behavior.
Kubernetes is an open-source system for automating deployment, scaling,
and management of containerized applications.
It is a complex environment with a huge amount of information regarding the state of cluster and events
processed during execution of pods lifecycle and health checking off all nodes and whole Kubernetes
cluster.
I do not have practice with Docker for Windows, so my point of view is based on Kubernetes with Linux containers
perspective.
To collect and analyze all of this information there are some tools like Fluentd, Logstash
and they are accompanied by tools such as Elasticsearch and Kibana.
Those cluster-level log aggregation can be realized using Kubernetes orchestration framework.
So we can expect that some running containers take care of gathering data and other containers
take care of other aspects of abstractions like analyzing and presentation layer.
Please notice that some solutions depend on cloud platform features where Kubernetes environment
is running. For example, GCP offers Stackdriver Logging.
We can mention some layers of log probes and analyses:
monitoring a pod
is the most rudimentary form of viewing Kubernetes logs.
You use the kubectl commands to fetch log data for each pod individually.
These logs are stored in the pod and when the pod dies, the logs die with them.
monitoring a node. Collected log for each node are stored in a JSON file. This file can get really large.
Node-level logs are more persistent than pod-level ones.
monitoring a cluster.
Kubernetes doesn’t provide a default logging mechanism for the entire cluster, but leaves this up
to the user and third-party tools to figure out. One approach is to build on the node-level logging.
This way, you can assign an agent to log every node and combine their output.
As you see, there is a niche on cluster level monitoring, so there is a reason to aggregate current logs and
offer a practical way to analyze and present results.
On the node level logging, popular log aggregator is Fluentd. It is implemented as a Docker container,
and it is run parallel with pod lifecycle. Fluentd does not store the logs themselves.
Instead, it sends their logs to an Elasticsearch cluster that stores the log information in a replicated set of nodes.
It looks like Elasticsearch is used as a data store of aggregated logs of working nodes.
This aggregator cluster consists of a pod with two instances of Elasticsearch.
The aggregated logs in the Elasticsearch cluster can be viewed using Kibana.
This presents a web interface, which provides a more convenient interactive method for querying the ingested logs
The Kibana pods are also monitored by the Kubernetes system to ensure they are running healthily and the expected
number of replicas are present.
The lifecycle of these pods is controlled by a replication-controller specification similar in nature to how the
Elasticsearch cluster was configured.
Back to your question. I'm pretty sure that the mentioned above also works with Kubernetes and Dockers
for Windows. From the other hand, I think the cloud platform or the Linux premise environment
is a natural space to live for them.
Answer was inspired by Cluster-level Logging of Containers with Containers and Kubernetes Logging articles.
I also like Configuring centralized logging from Kubernetes page and used An Introduction
to logging in Kubernetes at my beginning with Kubernetes.
I'm using docker on a bare metal server. I'm pretty happy with docker-compose to configure and setup applications.
Still some features are missing, like configuration management and monitoring maybe there are other solutions to solve this issues but I'm a bit overwhelmed by the feature set of Kubernetes and can't judge if it would help me here.
I'm also open for recommendations to solve the requirements separately:
Configuration / Secret management
Monitoring of my docker hostes applications (e.g. having some kind of dashboard)
Remot container control (SSH is okay with only one Server)
Being ready to scale my environment (based on multiple different Dockerized applications) to more than one server in future - already thinking about networking/service discovery issues with a pure docker-compose setup
I'm sure Kubernetes covers some of these features, but I have the feeling that it's too much focused on Cloud platforms where Machines are created on the fly (since I only have at most few bare metal Servers)
I hope the questions scope is not too broad, else please use the comment section and help me to narrow down the question.
Thanks.
I think the Kubernetes is absolutely much your requests and it is what you need.
Let's start one by one.
I have the feeling that it's too much focused on Cloud platforms where Machines are created on the fly (since I only have at most few bare metal Servers)
No, it is not focused on Clouds. Kubernates can be installed almost on any bare-metal platform (include ARM) and have many tools and instructions which can help you to do it. Also, it is easy to deploy it on your local PC using Minikube, which will prepare local cluster for you within VMs or right in your OS (only for Linux).
Configuration / Secret management
Kubernates has a powerful configuration and management based on special objects which can be attached to your containers. You can read more about configuration management in that article.
Moreover, some tools like Helm can provide you more automation and range of preconfigured applications, which you can install using a single command. And you can prepare your own charts for it.
Monitoring of my docker hostes applications (e.g. having some kind of dashboard)
Kubernetes has its own dashboard where you can get many kinds of information: current applications status, configuration, statistics and many more. Also, Kubernetes has great integration with Heapster which can be used with Grafana for powerful visualization of almost anything.
Remot container control (SSH is okay with only one Server)
Kubernetes controlling tool kubectl can get logs and connect to containers in the cluster without any problems. As an example, to connect a container "myapp" you just need to call kubectl exec -it myapp sh, and you will get sh session in the container. Also, you can connect to any application inside your cluster using kubectl proxy command, which will forward a port you need to your PC.
Being ready to scale my environment (based on multiple different Dockerized applications) to more than one server in future - already thinking about networking/service discovery issues with a pure docker-compose setup
Kubernetes can be scaled up to thousands of nodes. Or can have only one. It is your choice. Independent of a cluster size, you will get production-grade networking, service discovery and load balancing.
So, do not afraid, just try to use it locally with Minikube. It will make many of operation tasks more simple, not more complex.