I'm looking for the monitoring solution for the web application, deployed as a Swarm of Docker containers spread through 7-10 VMs. High level requirements are:
Configurable Web and REST interface to performance dashboard
General performance metrics on VM levels (CPU/Memory/IO)
Alerts when containers and/or VMs are going offline/restart
Possibility to drill down into containers process activity when needed
Host OS are CoreOS and Ubuntu
Any recommendations/best practices here?
NOTE: external Kibana installation is being used to collect application logs from Logstash agents deployed on VMs.
Based on your requirements, it sounds like Sematext Docker Agent would be a good fit. It runs as a tiny container on each Docker host and collects all host+containers metrics, events, and logs. It can parse logs, route them, blacklist/whitelist them, has container auto-discovery, and so on. In the end logs end up in Logsene and metrics and events end up in SPM, which gives you a single pane of glass sort of view into all your Docker ops bits, with alerting, anomaly detection, correlation, and so on.
I am currently evaluating bosun with scollector + cAdvisor support. Look ok so far.
Edit:
It should meet all the listed requirements and a little bit more. :)
Take a look at Axibase Time-Series Database / Google Cadvisor / collectd stack.
Disclosure: I work for the company that develops ATSD.
Deploy 1 Cadvisor container per VM to collect Docker container statistics. Cadvisor front-end allows you to view top container processes.
Deploy 1 ATSD container to ingest data from multiple Cadvisor instances.
Deploy collectd daemon on each VM to collect host statistics, configure collectd daemons to stream data into ATSD using write_atsd plugin.
Dashboards:
Host:
Container:
API / SQL:
https://github.com/axibase/atsd/tree/master/api#api-categories
Alerts:
ATSD comes with a built-in rule engine. You can configure a rule to watch when containers stops collecting data and trigger an email or system command.
Related
Trying to determine if cAdvisor + Prometheus is the OTHER option for monitoring openshift containers. Or if there is another combo that I can use natively from Prometheus.
cAdvisor is essentially built into K8S deployments so it's as native as you can get really. If you want to use additional software there are other ways to collect data with agents, but cAdvisor is quite well understood and efficient for doing this type of data collection. Prometheus also scrapes other K8S APIs aside from cAdvisor.
I'm doing a performance testing for a server, which deployed on docker containers (have three docker containers). I'm using jmeter to conduct performance testing.
How can I get the average resource utilization of three docker containers into jmeter during the testing time??
Depending on what you're trying to achieve from simplest to the most complex:
docker stats command
Special solutions for docker containers health monitoring like cAdvisor
Special solutions for JMeter for monitoring remote servers like:
PerfMon Plugin
SSHMon Plugin
APM tools like AppDynamics or Dynatrace
Is there a way to run a cAdvisor container in a Monitoring server and monitor docker containers in a separate server? Is there a command I can include when running cAdvisor?
Because I want to be able to monitor containers in a separate server but I’m not sure how to achieve that…
Any suggestions or shared knowledge would be very helpful. Thank you.
To take measurements from different machines, you will have to deploy cAdvisor to every separate server.
My source is:
A monitoring solution for Docker hosts, containers and containerized services
Extending the monitoring system
Dockprom Grafana dashboards can be easily extended to cover more then one Docker host. In order to monitor more hosts, all you need to do is to deploy a node-exporter and a cAdvisor container on each host and point the Prometheus server to scrape those.
On how to create and start a container by using the remote api, you can check this answer: How to use docker remote api to create container?
We are using Jenkins and docker for doing CI/CD. Our Jenkins is setup as master/slave style, where slaves are distributed across different data centers. when a new build needs to happen Jenkins master identifies a slave in one of the DC and spin up a ephemeral container and tear it down once done.
Due to firewall limitations, we only have about 10 ports open for the slaves in some of the DCs. for example Port Range: 8000 - 8010. In general docker uses the linux port ranges 32768 to 61000. The problem is Jenkins master can not talk to the containers if the host port is bound out of 8000 - 8010. Jenkins docker plugin has limitation where you can not bind multiple ports (may be I am wrong here). I would like to know if any way we can configure this at docker end or in Jenkins docker plugin.
After researching in many forums and talking to people, this is not possible or recommended even to try doing. The recommended implementation to overcome this issue is to move to Docker Swarm,
where you have only one virtual docker cloud
which takes care of spinning up containers behind the scenes and keep it ready for consumption even before the need arises. The configurations options are flexible.
Read more about Swarm here
https://docs.docker.com/swarm/
What will telemetry and monitoring tools show if I lunch in (2 options)
docker container
host system
Will they show cpu\memory and etc usage of container only or of host system?
What are best practise? Monitoring software in each container or in host system?
What you want to do is monitor both, the host(s) and the containers running on them. A good way to do that is run a container that collects all data on each docker host. That is how Sematext Docker Agent runs, for example -- it runs as a tiny container on each Docker host and collects all host+containers metrics, events, and logs. It then parses logs, can route them, blacklist/whitelist them, auto-discovery new containers, and so on. In the end logs end up in Logsene and metrics and events end up in SPM, which gives you a single pane of glass sort of view into all your Docker ops bits, with alerting, anomaly detection, correlation, and so on. I hope this helps and points you in the right direction.
The results should be exactly the same, because Docker containers are sharing their resources (unlike virtual machines).
Putting an agent in your containers is not advisable, not just for performance reasons, but it is an anti-pattern in the Docker world, where each container should run a single process. Better is to run a monitoring agent on the host or in a separate container that can be configured to extract metrics from the other containers. This is the way we work at CoScale. If you are interested, have a look at our solution for monitoring Docker.