I’m testing docker swarm on a multi node cluster.
version: 20.10.7
The point is that if I create a service with docker service create and then I join nodes everything works (I use --with-registry-auth, master node is logged in to a private registry on AWS) , it means applications are replicated on nodes with image pull and containers start.
I kill the nodes manually and scale the service to 0 with:
docker service scale myserv=0
then I when I start a new node, join it to the cluster and try scale up, the image on the node is not pulled down, it says “no such image”,
that’s strange since if I re-create the service it is able to pull the image on nodes. It is like docker service scale doesn’t login to the remote registry in the nodes.
Any tips to solve this out? it would be nice to add nodes/remove nodes and have containers scaled automatically as from the scale istruction of the service I've created.
Thanks
we do use docker swarm in our production extensively, we use ecr credential helper, check out the link amazon-ecr-credential-helper
Link to github : git hub project link
As described you can have a startup script on the nodes to store your credentials
Related
When i would like to join a node by selecting etcd, Controle Plane and Worker in rancher UI, i got this error:
Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [10.xxx.xxx.36]
Click here to see the screenshot
So Rancher it could not download the remind docker containers (like etcd, kubectl..) automatically since the docker images should be preceded by the proxy: <proxy_url>
example: docker pull <proxy_url>/ubuntu for downloading ubuntu images.
Any help to resolve this would be appreciated. Thank you in advance!
You can define a private registry that Rancher should use to build downstream Kubernetes clusters with by setting the system-default-registry parameter in the "Settings" section of the Rancher UI. Then when you launch clusters, it should use this registry to fetch the images. This assumes you have already copied the images needed to this repo (example of how to do that).
Since you already created this cluster, you'll need to regenerate the docker run command and reapply to the node.
I have setup a 3 node cluster (with no Internet access) with 1 manager and 2 worker-nodes using the standard swarm documentation.
How does the swarm manager in swarm mode know about the images present in worker nodes?
Lets say I have image A in worker-node-1 and image B in worker-node-2 and no images in the manager-node.
Now how do I start container for image A using the manager?
Will it start in manager or node-1?
When I query manager for the list of images will it give the whole list with A and B in it?
Does anyone know how this works?
I couldn’t get the details from the documentation.
Docker Swarm manager node may to be a worker one by the second role but not strictly necessary.
Image deployment policy is mapped via docker-compose.yml which has an information like target nodes, networks, hostnames, volumes, etc. in relation of particular service. So, it will start either in specified node or in emptiest default one.
Swarm manager communicates with the worker nodes via Docker networks:
When you initialize a swarm or join a Docker host to an existing swarm, two new networks are created on that Docker host:
an overlay network called ingress, which handles control and data
traffic related to swarm services. When you create a swarm service and
do not connect it to a user-defined overlay network, it connects to
the ingress network by default
a bridge network called
docker_gwbridge, which connects the individual Docker daemon to the
other daemons participating in the swarm.
Reference
During Swarm deployment, the images of it's services are being propagated to worker nodes according to their deployment policy.
The manager node will contain images once the node is the worker one too (correct me, if it won't).
The default configuration with swarm mode is to pull images from a registry server and use pinning to reference a unique hash for those images. This can be adjusted, but there is no internal mechanism to distribute images within a cluster.
For an offline environment, I'd recommend a stand alone registry server accessible to the cluster. You can even run it on the cluster. Push your image there, and point your server l services to the registry for their images to pull. See this doc for details on running a stand alone registry, or any of the many 3rd party options (e.g. Harbor): https://docs.docker.com/registry/
The other option is to disable the image pinning, and manually copy images to each of your swarm nodes. You need to do this in advance of deploying any service changes. You'll also lose the benefit of reused image layers when you manually copy them. Because of all this issues it creates, overhead to manage, and risk of mistakes, I'd recommend against this option.
Run the docker stack deploy command with --with-registry-auth that will give the Workers access to pull the needed image
By default Docker Swarm will pull the latest image from registry when deploying
I am using Docker version 17.12.1-ce.
I have set up a swarm with two nodes, and I have a stack running on the manager, while I am to instantiate new nodes on the worker (not within a service, but as stand-alone containers).
So far I have been unable to find a way to instantiate containers on the worker specifically, and/or to verify that the new container actually got deployed on the worker.
I have read the answer to this question which led me to run containers with the -e option specifying constraint:Role==worker, constraint:node==<nodeId> or constraint:<custom label>==<value>, and this github issue from 2016 showing the docker info command outputting just the information I would need (i.e. how many containers are on each node at any given time), however I am not sure if this is a feature of the stand-alone swarm, since docker info only the number of nodes, but no detailed info for each node. I have also tried with docker -D info.
Specifically, I need to:
Manually specify which node to deploy a stand-alone container to (i.e. not related to a service).
Check that a container is running on a specific swarm node, or check how many containers are running on a node.
Swarm commands will only care/show service-related containers. If you create one with docker run, then you'll need to use something like ssh node2 docker ps to see all containers on that node.
I recommend you do your best in a Swarm to have all containers as part of a service. If you need a container to run on nodeX, then you can create a service with a "node constraint" using labels and constraints. In this case you could restrict the single replica of that service to a node's hostname.
docker service create --constraint Node.Hostname==swarm2 nginx
To see all tasks on a node from any swarm manager:
docker node ps <nodename_or_id>
Similarly to the user in this question: How do I run private docker images on Google Container Engine:
I'd like to non-dockerhub docker images in GKE.
I would prefer to not have to explicitly prefix my images with a docker IP address.
This is in a live cluster, so re-installing nodes isnt really a good idea.
Is there a way I can fire up new nodes in the cluster with a configuration that adds a new default search path for a registry to the docker daemons on the kubelets, or similar workaround.
I have a swarm setup which has around 6 nodes. Whenever I execute a docker run or docker pull command from the swarm manager it downloads the new image on all the swarm nodes.
This is creating data redundancy and choking my network.
Is there any way I can avoid this ?
Swarm Nodes need Images available to them by design. That will help swarm to start the container on an available node immediately when current node hosting the container crashes or current hosting node goes into maintenance (Drain Mode).
On the other hand docker Images will be pulled one time only, and you can use them until you upgrade your service.
Another one, Docker is designed for microservices, If you Image getting too large, Maybe you should try to cut it down to multiple containers.