Docker zero down time - docker

I want to have to achieve zero downtime. I have created a service with 5 replicas and deploy into the stack then I want to update the existing service with some new changes and deploy into the stack with zero downtime.
version: "3"
services:
hello-html:
image: hello-html:latest
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- 80:80
networks:
mynetwork:
driver: overlay

Use Kubernetes container as a services (CAAS) to do update the application with zero downtime

Use docker service update - refer to https://docs.docker.com/engine/reference/commandline/service_update/ for syntax.
And use docker service ls to get a list of your services and know which one to update.

I needed to set up the following options for my services:
--update-order start-first
--update-delay 1m # depends how long your tasks needs untill it's ready
I also had to implement a HealthCheck for my api with:
start period 1m
This way docker doesn't redirect traffic before the task is ready.
For example:
HEALTHCHECK --interval=5m --timeout=3s --start-period=1m \
CMD curl -f http://localhost/ || exit 1

Related

Docker swarm container replicas are running but not able to hit the same in browser or with curl

Am following docker documentation and trying to create replica of nodes using the link --> https://docs.docker.com/get-started/part3/
I have followed the steps as mentioned in the document but am not able to get the expected application response on curl or browser.
docker decompose file looks like below :
version: "3"
services:
web:
image: vigneshnithin23/restaurant:latest
deploy:
replicas: 2
placement:
constraints: [node.role == manager]
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "8090:8090"
networks:
- webnet
networks:
webnet:
As per the documentation they are able to curl the URL. whereas am not able to do the same.
I had two IP address and initialised docker swarm with --advertise-addr - First wlan address
if I run the same code on single container am able to get the desired result.
I went through the question in below link, which was asked earlier but that didn't have proper answer
Website available in standalone container, not in swarm
Any help would be appreciated.
You need to ask yourself whether your SpringBoot app would be able to
run in that constrained environment of 50% usage of the CPU per second
and 50M.
When you run that as a single container[9ms for startup] it is running in a non-constrained environment while in your stack file you are restricting it. So when you spin up the service it takes a longer time to warm up and get started.
Here is a screenshot of the same app by just changing the replica set to '1' and increasing the memory to 100M it takes around 900ms to serve the application & serve the response.
So try to fine tune those settings for resource constraints.

Docker swarm does not distribute the container in the cluster

I have a two servers to use in a Docker cluster Swarm(test only), one is a Manager and other is a Worker, but running the command docker stack deploy --compose-file docker-compose.yml teste2 all the services is run in the manager and the worker not receive the containers to run, for some reason the Swarm is not achieving distributing the services in the cluster and running all in manager server.
Will my docker-compose.yml be causing the problem or might it be a network problem?
Here are some settings:
Servers CentOs 7, Docker version 18.09.4;
I executed the commands systemctl stop firewalld && systemctl disable firewalld to disable firewall;
I executed the command docker swarm join --token ... in the worker;
Result docker node ls:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
993dko0vu6vlxjc0pyecrjeh0 * name.server.manager Ready Active Leader 18.09.4
2fn36s94wjnu3nei75ymeuitr name.server.worker Ready Active 18.09.4
File docker-compose.yml:
version: "3"
services:
web:
image: testehello
deploy:
replicas: 5
update_config:
parallelism: 2
delay: 10s
restart_policy:
condition: on-failure
# placement:
# constraints: [node.role == worker]
ports:
- 4000:80
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- 8080:8080
stop_grace_period: 1m30s
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
webnet:
I executed the command docker stack deploy --compose-file docker-compose.yml teste2
In the docker-compose.yml I commented the parameters placement and constraints because they did not work and did not start the containers on the servers, without it the containers are started in the manager. Through the Visualizer all appear in the manager.
I think the images are not accessible from a worker node, that is why they not receive containers, try to use this guide by docker https://docs.docker.com/engine/swarm/stack-deploy/
P.S. I think you solved it already, but just in case.

Failed to allocate gateway with docker stack

I'm trying to execute the tutorial from the official documentation. It works fine except with Services.
When I start 5 instances of the container (with docker stack command), the containers are not starting and I get this error:
"failed to allocate gateway"
$ docker stack ps getstartedlab
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
imb6vgifjvq7 getstartedlab_web.1 seb/docker-whale:1.1 ns3553081.ip-XXX-YYY-ZZZ.eu Ready Rejected 4 seconds ago "failed to allocate gateway (1…"
ulm1tqdhzikd \_ getstartedlab_web.1 seb/docker-whale:1.1 ns3553081.ip-XXX-YYY-ZZZ.eu Shutdown Rejected 9 seconds ago "failed to allocate gateway (1…"
...
The docker-compose.yml contains
version: "3"
services:
web:
image: seb/docker-whale:1.1
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "80:80"
networks:
- webnet
networks:
webnet:
to start containers I'm using the command:
$ docker stack deploy -c docker-compose.yml getstartedlab
I can start without any issue one instance of the container with the command:
$ docker run -p 80:80 seb/docker-whale:1.1
Any idea why it's not working? How can I get more details on the error?
Thanks for your help.
Answer from a beginner: Same here (version 1.13.1), the message vanished when I changed ports "80:80" to "8080:80". Port 80 was used by the host of the docker machine.

How to specify Memory & CPU limit in docker compose version 3

I am unable to specify CPU and memory limitation for services specified in version 3.
With version 2 it works fine with mem_limit & cpu_shares parameters under the services. But it fails while using version 3, putting them under deploy section doesn't seem worthy unless I am using swarm mode.
Can somebody help?
version: "3"
services:
node:
build:
context: .
dockerfile: ./docker-build/Dockerfile.node
restart: always
environment:
- VIRTUAL_HOST=localhost
volumes:
- logs:/app/out/
expose:
- 8083
command: ["npm","start"]
cap_drop:
- NET_ADMIN
- SYS_ADMIN
I know the topic is a bit old and seems stale, but anyway I was able to use these options:
deploy:
resources:
limits:
cpus: '0.001'
memory: 50M
when using 3.7 version of docker-compose
What helped in my case, was using this command:
docker-compose --compatibility up
--compatibility flag stands for (taken from the documentation):
If set, Compose will attempt to convert deploy keys in v3 files to
their non-Swarm equivalent
Think it's great, that I don't have to revert my docker-compose file back to v2.
deploy:
resources:
limits:
cpus: '0.001'
memory: 50M
reservations:
cpus: '0.0001'
memory: 20M
More: https://docs.docker.com/compose/compose-file/compose-file-v3/#resources
In you specific case:
version: "3"
services:
node:
image: USER/Your-Pre-Built-Image
environment:
- VIRTUAL_HOST=localhost
volumes:
- logs:/app/out/
command: ["npm","start"]
cap_drop:
- NET_ADMIN
- SYS_ADMIN
deploy:
resources:
limits:
cpus: '0.001'
memory: 50M
reservations:
cpus: '0.0001'
memory: 20M
volumes:
- logs
networks:
default:
driver: overlay
Note:
Expose is not necessary, it will be exposed per default on your stack network.
Images have to be pre-built. Build within v3 is not possible
"Restart" is also deprecated. You can use restart under deploy with on-failure action
You can use a standalone one node "swarm", v3 most improvements (if not all) are for swarm
Also Note:
Networks in Swarm mode do not bridge. If you would like to connect internally only, you have to attach to the network. You can 1) specify an external network within an other compose file, or have to create the network with --attachable parameter (docker network create -d overlay My-Network --attachable)
Otherwise you have to publish the port like this:
ports:
- 80:80
Docker Compose v1 does not support the deploy key. It's only respected when you use your version 3 YAML file in a Docker Stack.
This message is printed when you add the deploy key to you docker-compose.yml file and then run docker-compose up -d
WARNING: Some services (database) use the 'deploy' key, which will be
ignored. Compose does not support 'deploy' configuration - use docker stack deploy to deploy to a swarm.
The documentation (https://docs.docker.com/compose/compose-file/#deploy) says:
Specify configuration related to the deployment and running of
services. This only takes effect when deploying to a swarm with docker
stack deploy, and is ignored by docker-compose up and docker-compose
run.
Nevertheless you can use Docker Compose v2. Given the following Docker composition you can use the deploy key to limit your containers resources.
version: "3.9"
services:
database:
image: mariadb:10.10.2-jammy
container_name: mydb
environment:
MYSQL_ROOT_PASSWORD: root_secret
MYSQL_DATABASE: mydb
MYSQL_USER: myuser
MYSQL_PASSWORD: secret
TZ: "Europe/Zurich"
MARIADB_AUTO_UPGRADE: "true"
tmpfs:
- /var/lib/mysql:rw
ports:
- "127.0.0.1:3306:3306"
deploy:
resources:
limits:
cpus: "4.0"
memory: 200M
networks:
- mynetwork
When you run docker compose up -d (Note: in version 2 of Docker Compose you call the docker binary at not the docker-compose python application) and then inspect the resources you see that the memory is limited to 200 MB. The CPU limit is not exposed by docker stats.
❯ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
2c71fb8de607 mydb 0.04% 198MiB / 200MiB 99.02% 2.67MB / 3.77MB 70.6MB / 156MB 18
This is possible with version >= 3.8. Here is an example using docker-compose >= 1.28.x :
version: '3.9'
services:
app:
image: nginx
cpus: "0.5"
mem_reservation: "10M"
mem_limit: "250M"
Proof of it working (see the MEM USAGE) column :
The expected behavior when reaching memory limit is the container getting killed. In this case, whether set restart: always or adjust your app code.
Limits and restarts settings in Docker compose v3 should now be set using (restart: always is also deprecated):
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
I have other experiences, maybe somebody can explain this.
Maybe this is bug(i think this is a feature), but, I am able to use deployments limits (memory limits) in docker-compose without swarm, hovever CPU limits doesn't work but replication does.
$> docker-compose --version
docker-compose version 1.29.2
$> docker --version
Docker version 20.10.12
version: '3.2'
services:
limits-test:
image: alexeiled/stress-ng
command: [
'--vm', '1', '--vm-bytes', '20%', '--vm-method', 'all', '--verify', '-t', ' 10m', '-v'
]
deploy:
resources:
limits:
cpus: '0.50'
memory: 1024M
Docker stats
b647e0dad247 dc-limits_limits-test_1 0.01% 547.1MiB / 1GiB 53.43% 942B / 0B 0B / 0B 3
Edited, thx #Jimmix
I think there is confusion here over using docker-compose and docker compose (with a space). You can install the compose plugin using https://docs.docker.com/compose/install if you don't already have it.
Here is an example compose file just running Elasticsearch
version: "3.7"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
restart: always
ports:
- "9222:9200"
deploy:
resources:
limits:
cpus: "4"
memory: "2g"
environment:
- "node.name=elasticsearch"
- "bootstrap.memory_lock=true"
- "discovery.type=single-node"
- "xpack.security.enabled=false"
- "ingest.geoip.downloader.enabled=false"
I have it in a directory called estest the file is called es-compose.yaml. The file sets CPU and memory limits.
If you launch using docker-compose e.g.
docker-compose -f es-compose.yaml up
Then look at docker stats you see
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e3b6253ee730 estest_elasticsearch_1 342.13% 32.39GiB / 62.49GiB 51.83% 7.7kB / 0B 27.3MB / 381kB 46
so the cpu and memory resource limits are ignored. During the launch you see the warning
WARNING: Some services (elasticsearch) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Which I think is what leads people to look at Docker stack/swarm. However if you just switch to using the newer docker compose now built in to the docker CLI https://docs.docker.com/engine/reference/commandline/compose/ e.g.
docker compose -f es-compose.yaml up
And look again at docker stats you see
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d062eda10ffe estest-elasticsearch-1 0.41% 1.383GiB / 2GiB 69.17% 8.6kB / 0B 369MB / 44MB 6
Therefore the limits have been applied.
This is better in my opinion than swarm as it still allows you to build containers as part of the compose project and pass environment easily via a file. I would recommend removing docker-compose and switching over to use the newer docker compose wherever possible.

Docker Swarm Service Clustering

I want to use docker-compose with Docker Swarm (I use docker version 1.13 and compose with version: '3' syntax).
Is each service reachable as a "single" service to the other services? Here is an simplified example to be clear:
version: '3'
services:
nodejs:
image: mynodeapp
container_name: my_app
ports:
- "80:8080"
environment:
- REDIS_HOST=my_redis
- REDIS_PORT=6379
deploy:
mode: replicated
replicas: 3
networks:
- my_net
command: npm start
redis:
image: redis
container_name: my_redis
restart: always
expose:
- 6379
deploy:
mode: replicated
replicas: 2
networks:
- my_net
networks:
my_net:
external: true
Let's say I have 3 VMs which are configured as a swarm. So there is one nodejs container running on each VM but there are only two redis container.
On the VM where no redis is running: Will my nodejs container know about the redis?
Addiitonal questions:
When I set replicas: 4 for my redis, I will have two redis container on one VM: Will this be a problem for my nodejs app?
Last question:
When I set replicas: 4 for my nodeapp: Will this even work because I now have exposed two times port 80?
The services have to be stateless. In the case of databases it is necessary to set the cluster mode in each instance, since they are statefull.
In the same order you asked:
One service does not see another service as if it is made of replicas. Nodejs will see a unique Redis, which will have one IP, no matter in which node its replicas are located. That's the beauty of Swarm.
Yes, you can have Nodejs in one node and Redis in another node and they will be visible to each other. That's what the manager does; make the containers "believe" they are running on the same machine.
Also, you can have many replicas in the same node without a problem; they will be perceived as a whole. In fact, they use the same volume.
And last, as implication of (1), there will be no problem because you are not actually exposing port 80 twice. Even having 20 replicas, you have a unique entrypoint to your service, a particular IP:PORT direction.

Resources