I have a service with just 1 instance on a docker swarm. It exposes a port to the machine.
I want docker swarm easily apply rolling update without downtime. Right now i am running docker service rm and then docker service create
Using just 1 instance / replica as i am using websockets.
What is best strategy and command i should use for rolling update of single instance with exposed port.
I tried the order: start-first with replicas: 1. With this setup, existing docker container is shutdown prior to startup the new docker container. So it seem that the order start-first or stop-first doesn't really matter, or the start-first didn't work as desire.
If however I set replicas: 2, then one of the existing docker container will be replaced with a new docker container, then after that one succeeds then the second docker container is replacing the other existing docker container
A way to achieve this is to use order:start-first in the update_config section of the docker service in the compose file. which will start a new container before terminating the old one when you are applying rolling update or deploying a newer version.
you can check here for more details about update_config :https://docs.docker.com/compose/compose-file/compose-file-v3/
Related
I have 3 container for single service that created with --scale option on docker-compose. when I tried to recreate them, all of containers stop and remove, and after that, docker start to recreate one by one.
how can I to complete this process one by one, for example, stop first container and recreate them , after container up complete, go next one.
Unfortunately docker-compose don't have this feature, but Docker Swarm does!
Just init your docker machine to swarm cluster with
docker swarm init
and then reconfigure your compose file and add rolling updates like so:
deploy:
replicas: 2
update_config:
parallelism: 2
delay: 10s
order: stop-first
Lets say I want to edit a config file for an NGINX Docker service that is replicated across 3 nodes.
Currently I list the services using docker service ls.
Then get the details to find a node running a container for that service using docker serivce ps servicename.
Then ssh to a node where one of the containers is running.
Finally, docker exec -it containername bash. Then I edit the config file.
Two questions:
Is there a better way to do this rather than ssh to a node running a container? Maybe there is a swarm or service command to do so?
If I were to edit that config file on one container would that change be replicated to the other 2 containers in the swarm?
The purpose of this exercise would be to edit configuration without shutting down a service.
You should not be exec'ing into containers to change their configuration, and so docker has not created an easy way to do this within Swarm Mode. You could use classic swarm to avoid the need to ssh into the other host, but I still don't recommend this.
The correct way to do this is to migrate your configuration file into a docker config entry. Version your config name. Then when you want to update it, you create a new version with the desired changes, and do a rolling update of your service to use that new configuration.
Unless the config is mounted from an external source like NFS, changes to one config in one container will not apply to other containers running on other nodes. If that config is stored locally inside your container as part of it's internal copy-on-write filesystem, then no changes from one container will be visible in any other container.
I am having a problem trying to implement the best way to add new container to an existing cluster while all containers run in docker.
Assuming I have a docker swarm, and whenever a container stops/fails for some reason, the swarm bring up new container and expect it to add itself to the cluster.
How can I make any container be able to add itself to a cluster?
I mean, for example, if I want to create a RabbitMQ HA cluster, I need to create a master, and then create slaves, assuming every instance of RabbitMQ (master or slave) is a container, let's now assume that one of them fails, we have 2 options:
1) slave container has failed.
2) master container has failed.
Usually, a service which have the ability to run as a cluster, it also has the ability to elect a new leader to be the master, so, assuming this scenerio is working seemlesly without any intervention, how would a new container added to the swarm (using docker swarm) will be able to add itself to the cluster?
The problem here is, the new container is not created with new arguments every time, the container is always created as it was deployed first time, which means, I can't just change it's command line arguments, and this is a cloud, so I can't hard code an IP to use.
Something here is missing.
Maybe trying to declare a "Service" in the "docker Swarm" level, will acctualy let the new container the ability to add itself to the cluster without really knowing anything the other machines in the cluster...
There are quite a few options for scaling out containers with Swarm. It can range from being as simple as passing in the information via a container environment variable to something as extensive as service discovery.
Here are a few options:
Pass in IP as container environment variable. e.g. docker run -td -e HOST_IP=$(ifconfig wlan0 | awk '/t addr:/{gsub(/.*:/,"",$2);print$2}') somecontainer:latest
this would set the internal container environment variable HOST_IP to the IP of the machine it was started on.
Service Discovery. Querying a known point of entry to determine the information about any required services such as IP, Port, ect.
This is the most common type of scale-out option. You can read more about it in the official Docker docs. The high level overview is that you set up a service like Consul on the masters, which you have your services query to find the information of other relevant services. Example: Web server requires DB. DB would add itself to Consul, the web server would start up and query Consul for the databases IP and port.
Network Overlay. Creating a network in swarm for your services to communicate with each other.
Example:
$ docker network create -d overlay mynet
$ docker service create –name frontend –replicas 5 -p 80:80/tcp –network mynet mywebapp
$ docker service create –name redis –network mynet redis:latest
This allows the web app to communicate with redis by placing them on the same network.
Lastly, in your example above it would be best to deploy it as 2 separate containers which you scale individually. e.g. Deploy one MASTER and one SLAVE container. Then you would scale each dependent on the number you needed. e.g. to scale to 3 slaves you would go docker service scale <SERVICE-ID>=<NUMBER-OF-TASKS> which would start the additional slaves. In this scenario if one of the scaled slaves fails swarm would start a new one to bring the number of tasks back to 3.
https://docs.docker.com/engine/reference/builder/#healthcheck
Docker images have a new layer for health check.
Use a health check layer in your containers for example:
RUN ./anyscript.sh
HEALTHCHECK exit 1 or (Any command you want to add)
HEALTHCHECK check the status code of command 0 or 1 and than result as
1. healthy
2. unhealthy
3. starting etc.
Docker swarm auto restart the unhealthy containers in swarm cluster.
I have some problem using docker swarm mode .
I want to have high availability with swarm mode.
I think I can do that with rolling update of swarm.
Something like this...
docker service update --env-add test=test --update-parallelism 1 --update-delay 10s 6bwm30rfabq4
However there is a problem.
My docker image have entrypoint. Because of this there is a little delay before the service(I mean docker container) is really up. But docker service just think the service is already running, because status of the container is 'Up'. Even the service still do some work on entrypoint. So some container return error when I try to connect the service.
For example, if I create docker service named 'test' and scale up to 4 with port 8080. I can access test:8080 on web browser. And I try to rolling update with --update-parallelism 1 --update-delay 10s options. After that I try to connect the service again.. one container return error.. Because Docker service think that container already run..even the container still doesn't up because of entrypoint. And after 10s another container return error.. because update is started and docker service also think that container is already up.
So.. Is there any solution to solve this problem?
Should I make some nginx settings for disconnect connection to error container and reconnect another one?
The HEALTHCHECK Dockerfile command works for this use case. You specify how Docker should check if the container is available, and it gets used during updates as well as checking service levels in Swarm.
There's a good article about it here: Reducing Deploy Risk With Docker’s New Health Check Instruction.
I have to create a huge number of Docker container on different hosts (e.g. 50 container each on 3 hosts). These container all have the same image, configuration etc. and only the network address and ID of each container should be different (so basically I want to create a huge virtual container network).
Is there a way to achieve this?
I have looked at technologies like Helios and Kubernetes but they seem to only deploy one container on each agent. I thought about just creating a lot of different jobs in Helios and then deploy each one of them to its agent, but that seems a little dirty to me.
This is exactly the type of use case that Kubernetes is well suited for.
You should use a Replica Set. When creating your Replica Set, you specify a template that tells the system how to create each container instance along with a desired number of replicas. It will then create that number of replicas in the available number of nodes in your cluster.
One caveat is that by default, Kubernetes will only allow you to have ~100 pods / node, but you can change this number with a command line flag if you need more.
For the Docker specific solution, you can use Swarm and Compose.
Create your Docker Swarm cluster of 3 nodes and change your environment to that Swarm. (The below assumes each host is listening on 2375, which is ok for a private network, but you'll want TLS setup and switch over to 2376 for more security.)
cat >cluster.txt <<EOF
node1:2375
node2:2375
node3:2375
EOF
docker run -d -P --restart=always --name swarm-manager \
-v ./cluster.txt:/cluster.txt \
swarm manage file:///cluster.txt
export DOCKER_HOST=$(docker port swarm-manager 2375)
Define your service inside of a docker-compose.yml, and then run docker-compose scale my-service=150. If your Swarm is setup with the default spread strategy, it will distribute them across the 3 hosts based on the number of containers running (or stopped) on each.
cat >docker-compose.yml <<EOF
my-app:
image: my-app
EOF
docker-compose scale my-app=150
Note that the downside of docker-compose over the other tools out there is that it doesn't correct for outages until you rerun it.