I am deploying my app via docker-compose, that have 2 services: server app and nginx.
So in CI I created following script of instructions:
docker-compose build # create new containers
docker-compose down # down old containers
docker-compose up -d # up new containers
But server app container has its own start up time, so right after app start I see 502 page, because server app is not yet ready to receive calls, but nginx is ready.
What I want to do is to preserve old containers running, during that build and up new containers, wait some time for server app to be ready and then somehow substitute them. So whole operation whould be seamless for users.
How can I do it?
It is not possible with docker-compose. But, yes it is possible with container orchestration tools like Kubernetes, the popular orchestration tools.
From Kubernetes official site :
Automated rollouts and rollbacks
Kubernetes progressively rolls out changes to your application or its
configuration, while monitoring application health to ensure it
doesn't kill all your instances at the same time. If something goes
wrong, Kubernetes will rollback the change for you. Take advantage of
a growing ecosystem of deployment solutions.
Self-healing
Restarts containers that fail, replaces and reschedules containers
when nodes die, kills containers that don't respond to your
user-defined health check, and doesn't advertise them to clients until
they are ready to serve.
Related
Is it possible to restart a container if another container fails and restarts?
I have a server container and multiple client containers, I want to have it that if the server container fails and restarts, that one of the client containers restarts as well.
I've already used the restart policies (always, on-failure etc.) but this would be linking two containers and triggering the restart of container A if container B restarts.
This question seems to be quite similar, if not duplicate, of this one.
TL;DR: There has been a shift from defining complex restart policies in docker/docker-compose superseded by explicitly checking for dependencies from within your service so it is deployment agnostic. Therefore, the recommendation is to create specific checks within the container that 'depends' on other services and crash properly when they are not met so that a simple restart: always policy is all that is needed.
Scenario:
There is a container running with image version 1.0 and exposed port 8080 on localhost 80. The new version of the image is available, and there is a need to switch those versions. No, any orchestration tool is running ( Kubernetes, OpenShift etc...).
Is it possible to start a container with version 1.1 make it run without a problem
Please, keep in mind that I don't want to keep it simple, no replication, etc.
Simply docker container with the binded port to localhost.
Questions:
1. Is it possible to switch exposing of port between containers without downtime?
2. If not, is there is any mechanism implemented with docker (free edition) to do such switch?
Without downtime, you'd need a second replica of the service up an running, and a proxy in front of that service that's listening to user requests and routing from one to the other. Both Swarm Mode and Kubernetes provide this capability with similar tools, the port being exposed is indirectly connected to the app via either an application reverse proxy, or some iptables rules and ipvs entries in the kernel.
Out of the box, recent versions of docker include support for Swarm Mode with nothing additional to install. You can run a simple docker swarm init to start a single node swarm cluster in less than a second. And then instead of docker-compose up you switch to docker stack deploy -c docker-compose.yml $stack_name to manage your projects with almost the same compose file. For swarm mode, you'll want to be on version 3 of the compose file syntax.
For a v3 syntax compose file in swarm mode that has no outage on an update, you'll want healthcheck's defined in your image to monitor the application and report back when it's ready to receive requests. Then you'll want a deploy section of the compose file to either have multiple replicas for HA, or at least configure a single replica to have a "start-first" policy to ensure the new service is up before stopping the old one. See the compose docs for settings to adjust: https://docs.docker.com/compose/compose-file/#update_config
For an application based reverse proxy in docker, I really do like traefik, but more to allow me to run multiple http based container services with a single port opened. This allows me to mapping requests based off the hostname/path/http header to the right container, while at the same time giving features to migrate between different versions with weighting of which backend to use so you can do more than a simple round-robin load balancing during an upgrade.
There is no mechanism native to Docker that would allow you replace one container with another with no interruption. On the other hand, the duration of the interruption can probably be measured in milliseconds; whether or not this is really an issue for you depends entirely on your application.
You can get the behavior you want by introducing a dynamic reverse proxy such as Traefik into your configuration. The proxy binds to host ports and handles requests from remote systems, then distributes those requests to one or more backend containers.
You can create and remove backend containers as you please, and as long as at least one is running your application will be available. For your specific use case, this means that you can start the new version of your application first, then retire the old one, all without any interruption in service.
I previously created a Flask server that spawns Docker containers using the Docker Python SDK. When a client hits a specific endpoint, the server would generate a container. It would maintain queues, and it would be able to kill containers that didn't respond to requests.
I want to migrate towards Kubernetes, but I am starting to think my current server won't be able to "spawn" jobs as pods automatically like in docker.
docker.from_env().containers.run('alpine', 'echo hello world')
Is Docker Swarm a better solution for this, or is there a hidden practice that is done in Kubernetes? Would the Kubernetes Python API be a logical solution for automatically generating pods and jobs, where the Flask server is a pod that manages other pods within the cluster?
'Kubectl run' is much like 'docker run' in that it will create a Pod with a container based on a docker image (e.g. How do i run curl command from within a Kubernetes pod). See https://kubernetes.io/docs/reference/kubectl/docker-cli-to-kubectl/ for more comparison. But what you run with k8s are Pods/Jobs that contain containers rather than running containers directly so this will add an extra layer of complexity for you.
Kubernetes is more about orchestrating services rather than running short-lived jobs. It has some features and can be used to run jobs but that isn't its central focus. If you're going in that direction you may want to look at knative (and knative build) or kubeless as what you describe sounds rather like the serverless concept. Or if you are thinking more about Jobs then perhaps brigade (https://brigade.sh). (For more see https://www.quora.com/Is-Kubernetes-suited-for-long-running-batch-jobs) If you are looking to run web app workloads that serve requests then note that you don't need to kill containers that fail to respond on k8s as k8s will monitor and restart them for you.
I don't know swarm well enough to compare. I suspect it would be a bit easier for you as it is aimed more centrally at docker (the k8s API is intended to support other runtimes) but perhaps somebody else can comment on that. Whether using swarm instead helps you will I guess depend on your motivations.
I have a couple of compose files (docker-compose.yml) describing a simple Django application (five containers, three images).
I want to run this stack in production - to have the whole stack begin on boot, and for containers to restart or be recreated if they crash. There aren't any volumes I care about and the containers won't hold any important state and can be recycled at will.
I haven't found much information on using specifically docker-compose in production in such a way. The documentation is helpful but doesn't mention anything about starting on boot, and I am using Amazon Linux so don't (currently) have access to Docker Machine. I'm used to using supervisord to babysit processes and ensure they start on boot up, but I don't think this is the way to do it with Docker containers, as they end up being ultimately supervised by the Docker daemon?
As a simple start I am thinking to just put restart: always on all my services and make an init script to do docker-compose up -d on boot. Is there a recommended way to manage a docker-compose stack in production in a robust way?
EDIT: I'm looking for a 'simple' way to run the equivalent of docker-compose up for my container stack in a robust way. I know upfront that all the containers declared in the stack can reside on the same machine; in this case I don't have need to orchestrate containers from the same stack across multiple instances, but that would be helpful to know as well.
Compose is a client tool, but when you run docker-compose up -d all the container options are sent to the Engine and stored. If you specify restart as always (or preferably unless-stopped to give you more flexibility) then you don't need run docker-compose up every time your host boots.
When the host starts, provided you have configured the Docker daemon to start on boot, Docker will start all the containers that are flagged to be restarted. So you only need to run docker-compose up -d once and Docker takes care of the rest.
As to orchestrating containers across multiple nodes in a Swarm - the preferred approach will be to use Distributed Application Bundles, but that's currently (as of Docker 1.12) experimental. You'll basically create a bundle from a local Compose file which represents your distributed system, and then deploy that remotely to a Swarm. Docker moves fast, so I would expect that functionality to be available soon.
You can find in their documentation more information about using docker-compose in production. But, as they mention, compose is primarily aimed at development and testing environments.
If you want to use your containers in production, I would suggest you to use a suitable tool to orchestrate containers, as Kubernetes.
If you can organize your Django application as a swarmkit service (docker 1.11+), you can orchestrate the execution of your application with Task.
Swarmkit has a restart policy (see swarmctl flags)
Restart Policies: The orchestration layer monitors tasks and reacts to failures based on the specified policy.
The operator can define restart conditions, delays and limits (maximum number of attempts in a given time window). SwarmKit can decide to restart a task on a different machine. This means that faulty nodes will gradually be drained of their tasks.
Even if your "cluster" has only one node, the orchestration layer will make sure your containers are always up and running.
You say that you use AWS so why don't you use ECS which is built for what you ask. You create an application which is the pack of your 5 containers. You will configure which and how many instances EC2 you want in your cluster.
You just have to convert your docker-compose.yml to the specific Dockerrun.aws.json which is not hard.
AWS will start your containers when you deploy and also restart them in case of crash
I have a docker container that I want to deploy to a CoreOS cluster that has to download my app from a git repo.
Let's say the app container runs nginx / nodejs
How should I update it?
If i submit the container and start it, that works the first time. But the second time I'll have to stop/start the container with fleetctl then I'll obviously have downtime. Should I start up new containers that are derived from that container?
Here's a complete walkthrough on exactly such a scenario:
http://coreos.com/blog/zero-downtime-frontend-deploys-vulcand.html
Instead of pulling down your application from github inside your container, you should bake your application code inside your container/image. Your container should start its services within a few seconds. To achieve zero downtime you should keep the old container running until your new container has started and is ready to accept new connections. You could do this by separating nginx into its own container and keep it running all the time.