I am new to Docker container and my question is how to monitor a process that is running inside a container. For Example, I have a container running apache in it. how would I know if apache process inside container got killed but my container is still running.
How we will ensure specific process inside the container is running,if that process goes down how we will get alert ?
The Dockerfile reference has the answer:
https://docs.docker.com/engine/reference/builder/
More specifically, the HEALTHCHECK directive:
https://docs.docker.com/engine/reference/builder/#healthcheck
Essentially, when your container's entrypoint fails, the container dies:
https://docs.docker.com/engine/reference/builder/#entrypoint
But, in any case, a process running inside a container is also visible from the host's process list, so you can safely use the output of ps aux| grep httpd to monitor your apache's PIDs.
In production , you don't just use docker run , you need to use some container orchestrator like kubernetes where you define the health checks such as liveness and readiness probes and the orchestrator will take care of the rest , it will restart the container if apache fails for some reason.
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
Related
I need some clarification in regards to using HEALTHCHECK on a docker service.
Context:
We are experimenting with a multi-node mariadb cluster and by utilizing HEALTHCHECK we would like the bootstrapping containers to remain unhealthy until bootstrapping is complete. We want this so that front-end users don’t access that particular container in the service until it is fully online and sync’d with the cluster. The issue is that bootstrapping relies on the network between containers in order to do a state transfer and it won’t work when a container isn’t accessible on the network.
Question:
When a container’s status is either starting or unhealthy does HEALTHCHECK completely kill network access to and from the container?
As an example, when a container is healthy I can run the command getent hosts tasks.<service_name>
inside the container which returns the IP address of other containers in a service. However, when the same container is unhealthy that command does not return anything… Hence my suspicion that HEALTHCHECK kills the network at the container level (as opposed to at the service/load balancer level) if the container isn’t healthy.
Thanks in advance
I ran some more tests and found my own answer. Basically docker does not kill container networking when it is either in the started or unhealthy phase. The reason getent hosts tasks.<service_name> command does not work during those phases is that that command goes back to get the container IP address through the service which does not have the unhealthy container(s) assigned to it.
I'm using Docker to run a java REST service in a container. If I were outside of a container then I might use a process manager/supervisor to ensures that the java service restarts if it encounters a strange one-off error. I see some posts about using supervisord inside of containers but it seems like they're focused mostly on running multiple services, rather than just keeping one up.
What is the common way of managing services that run in containers? Should I just be using some built in Docker stuff on the container itself rather than trying to include a process manager?
You should not use a process supervisor inside your Docker container for a single-service container. Using a process supervisor effectively hides the health of your service, making it more difficult to detect when you have a problem.
You should rely on your container orchestration layer (which may be Docker itself, or a higher level tool like Docker Swarm or Kubernetes) to restart the container if the service fails.
With Docker (or Docker Swarm), this means setting a restart policy on the container.
I am running a docker container which contains a node server. I want to attach to the container, kill the running server, and restart it (for development). However, when I kill the node server it kills the entire container (presumably because I am killing the process the container was started with).
Is this possible? This answer helped, but it doesn't explain how to kill the container's default process without killing the container (if possible).
If what I am trying to do isn't possible, what is the best way around this problem? Adding command: bash -c "while true; do echo 'Hit CTRL+C'; sleep 1; done" to each image in my docker-compose, as suggested in the comments of the linked answer, doesn't seem like the ideal solution, since it forces me to attach to my containers after they are up and run the command manually.
This is by design by Docker. Each container is supposed to be a stateless instance of a service. If that service is interrupted, the container is destroyed. If that service is requested/started, it is created. If you're using an orchestration platform like k8s, swarm, mesos, cattle, etc at least.
There are applications that exist to represent PID 1 rather than the service itself. But this goes against the design philosophy of microservices and containers. Here is an example of an init system that can run as PID 1 instead and allow you to kill and spawn processes within your container at will: https://github.com/Yelp/dumb-init
Why do you want to reboot the node server? To apply changes from a config file or something? If so, you're looking for a solution in the wrong direction. You should instead define a persistent volume so that when the container respawns the service would reread said config file.
https://docs.docker.com/engine/admin/volumes/volumes/
If you need to restart the process that's running the container, then simply run a:
docker restart $container_name_or_id
Exec'ing into a container shouldn't be needed for normal operations, consider that a debugging tool.
Rather than changing the script that gets run to automatically restart, I'd move that out to the docker engine so it's visible if your container is crashing:
docker run --restart=unless-stopped ...
When a container is run with the above option, docker will restart it for you, unless you intentionally run a docker stop on the container.
As for why killing pid 1 in the container shuts it down, it's the same as killing pid 1 on a linux server. If you kill init/systemd, the box will go down. Inside the namespace of the container, similar rules apply and cannot be changed.
I have a process running inside a container. I want to know this process's pid in docker host namespace from inside the container. I have a docker client installer inside the container from which I can execute a few commands but I do not have full privileges since I'm doing so from inside the container. Is there a way to find this pid?
The purpose of this approach was to make sure that only one instance of a process is running at time on a docker host. So the idea is to store the pid of the process at docker host level, so that it can be checked before starting a new instance.
I found another way to do so by using container id. You can do docker ps from inside the container and it will list all the containers running on docker host. Moreover container id does not change with namespace unlike pid. That solves my problem.
Is it possible to postpone the startup of a container based the availability of a separate HTTP service. For example, only start the container if port 8080 is running?
That sort of application-level service check isn't available in docker-compose. You would need to implement the necessary logic in your docker images.
For example, if you have something that depends on a web service, you could have your CMD run a script that does something like:
while ! curl -sf http://servicehost:8080/; do
sleep 1
done
exec myprogram
Another option is to set a restart policy of always on your containers, and have them fail if the target service isn't available. Docker will continue to restart your container until it keeps running.