I have a Marathon-Mesos-Docker in an inconsistent state where Marathon says the task is in a Waiting state, Mesos keeps trying to restart the task, but the container is actually running in Docker.
Anyone else seen this and, if so, what you did to fix it?
--John
Okay, I was able to figure it out. The Mesos agent crashed and, therefore, it could not send zookeeper state changes to Marathon. Once I restarted the mesos agent, the state of the Marathon task switched to running
Related
Documentation says:
unless-stopped Similar to always, except that when the container is stopped (manually or otherwise), it is not restarted even after Docker daemon restarts.
Ok. I understand what manually means: docker stop container_name. But what or otherwise stands for?
The paragraph after the table clarifies (emphasis mine):
configures it to always restart unless it is explicitly stopped or Docker is restarted.
One example is if the host reboots. Containers will be implicitly stopped (the container metadata and filesystems exist but the main container process does not), and at this point restart policies apply as well.
Event
no
on-failure
unless-stopped
always
docker stop
Stopped
Stopped
Stopped
Stopped
Host reboot
Stopped
Stopped
Stopped
Restarted
Process exits (code=0)
Stopped
Stopped
Restarted
Restarted
Process exits (codeā 0)
Stopped
Restarted
Restarted
Restarted
The documentation hints that this also applies if the Docker daemon is restarted, but this is a somewhat unusual case. My memory is that this event frequently seems to not affect running containers at all.
I'm having trouble with docker where docker ps won't return and is stuck.
I found that doinng docker service restart something like
sudo service docker restart (https://forums.docker.com/t/what-to-do-when-all-docker-commands-hang/28103/4)
However I'm worried if it will kill all the running containers? (I guess the service do provide service so that docker containers can run?)
In the default configuration, your assumption is correct: If the docker daemon is stopped, all running containers are shut down.. But, as outlined on the link, this behaviour can be changed on docker >= 1.12 by adding
{
"live-restore": true
}
to /etc/docker/daemon.json. Crux: the daemon must be restarted for this change to take effect. Please take note of the limitations of live reload, e.g. only patch version upgrades are supported, not major version upgrades.
Another possibility is to define a restart policy when starting a container. To do so, pass one of the following values as value for the command line argument --restart when starting the container via docker run:
no Do not automatically restart the container. (the default)
on-failure Restart the container if it exits due to an error, which manifests
as a non-zero exit code.
always Always restart the container if it stops. If it is manually stopped,
it is restarted only when Docker daemon restarts or the container
itself is manually restarted.
(See the second bullet listed in restart policy details)
unless-stopped Similar to always, except that when the container is stopped
(manually or otherwise), it is not restarted even after Docker
daemon restarts.
For your specific situation, this would mean that you could:
Restart all containers with --restart always (more on that further below)
Re-configure the docker daemon to allow for live reload
Restart the docker daemon (which is not yet configured for live reload, but will be after this restart)
This restart would shut down and then restart all your containers once. But from then on, you should be free to stop the docker daemon without your containers terminating.
Handling major version upgrades
As mentioned above, live reload cannot handle major version upgrades. For a major version upgrade, one has to tear down all running containers. With a restart policy of always, however, the containers will be restarted after the docker daemon is restarted after the upgrade.
We are running a scheduling engine with docker, chronos & mesos.
Running 2 mesos slaves on each node. Sometimes, too many Jobs gets executed on each node and docker becomes unresponsive and docker gets corrupted on rebooting the server. Is there anything wrong with the setup? Not sure, why docker hangs and gets corrupted on reboot?
Thanks
Running Docker containers won't work properly because restarting one agent
will cause Docker containers managed by the other agent to be deleted.
Check out --cgroups_root flag in
https://github.com/apache/mesos/blob/master/docs/configuration/agent.md
This flag only applies to MesosContainerizer (can be used to launch Docker
containers).
What is the correct way to restart WSO2 products from the command line when running in a Docker container?
Each management console provides a 'Forced Restart' function, which gracefully restarts the product. However, I've not been able to determine what this function actually does.
I've attempted to run wso2server.sh restart from within the container, but this does not restart the ENTRYPOINT process in Docker. It has also crashed my containers on several occasions.
Forced restart and gracefully restart also should work. However you can restart the docker container for your purpose.For more information on running wso2 products on docker you can follow this guide.
Hope this helps.
I'm running a custom reposository from Docker Hub on Tutum and trying to start a service. I get a Service Start, but then 1-2 seconds, it receives a Container status die command. I can't figure out why.
Received event 'die' from node 31b3f894-a0b9-494e-bca0-9212b642dd15.node.dockerapp.io for container api-1: Running
Changing container state from 'Running' to 'Stopped'
Checking triggers
Container has no triggers
I got an answer back from Docker Cloud (formerly Tutum) support.
When the process started inside a container exits for whatever reason,
the container is then considered "exited". Docker cloud shows this
service as "stopped" once it sees the container is no longer running.
So this particular container is trying to connect to a database (which I don't have connected again) so it fails, therefore that is why it tells the service to shut down.