I deploy a docker swarm with 6 nodes. I built some images and I am trying to add them as services to the swarm. I have 5 microservices. When I run the on one host with docker-compose everything works fine. I run this command docker service create rate --with-registry-auth and I get the following message.
image rate:latest could not be accessed on a registry to record
its digest. Each node will access rate:latest independently,
possibly leading to different nodes running different
versions of the image.
yyf9m49xw3enwano1scr55ufc
overall progress: 0 out of 1 tasks
1/1: No such image: rate:latest
I run docker images and the rate image is appeared. rate is the repository name. I also tried with the image id but didn't worked. The only images that I can add to swarm is images that is public.
There is an issue (https://github.com/moby/moby/issues/35187) on the Moby project about that.
If you tried already to set --with-registry-auth but didn't solved it, you should manually login to each cluster worker node and pull the Docker images.
image rate:latest could not be accessed on a registry to record
its digest. Each node will access rate:latest independently,
possibly leading to different nodes running different
versions of the image.
This error indicates you are trying to run an image that was never pushed to a registry. Push your images to a registry first. And then you can run them on any node in the cluster (which will pull any missing images from that registry). If the registry requires authentication to pull the image, then run docker stack deploy --with-registry-auth ..., but you must first push the image, and specific the pushed image name (which will not be rate:latest since you do not have access to push to the official library on Docker Hub).
Related
I've set up a Docker Swarm consisting of two VMs on my local network, (1 manager, 1 worker). In the manager node, I created a private registry service and I want to deploy a number of locally built images in my local dev machine (which is not in the swarm) to that registry. The Swarm docs and the dozens of examples I've read in the Internet seem not to go beyond the basics, running commands inside the manager node, building, tagging and pushing images from the manager's local cache to the registry in that same node, and I have that uneasy feeling that I'm missing something right on my face.
I see that my machine could simply join the swarm as a manager, owning the registry. The other nodes would automagically receive the updates and my problem would go away. But does this make sense for a production swarm setting, a cluster of nodes serving production code, depending on my dev's home machine - even as non-worker, manager-only?
Things I've tried:
Retagging my local image to <my.node.manager.ip>/my_app:1.0.0, followed by docker-compose push. I can see this does push the image to the manager's registry, but the service fails to start with the message "No such image: <my.node.manager.ip>/my_app:1.0.0"
Creating a context and, from my machine, run docker-compose --context my_context up --no-start. This (re)creates the image in the manager node's local cache, which I can then push to the registry, but it feels very unwieldy as a deploy process.
Should I run a remote script in the manager node to git pull my code and then do the build/push/docker stack deploy?
TL;DR What's the expected steps to deploy an image/app to a Docker Swarm from a local dev machine outside the swarm? Is this possible? Is this supported by Docker Swarm?
After reading a bit more on private registries and tags, I could finally wrap my head around the necessary tagging for my use case to work. My first approach was halfway right, but I had to change my deploy script so as to:
extract the image field from my docker-compose.yml (in the form localhost:5000/my_app:${MY_APP_VERSION-latest}, to circumvent the "No such image" error)
create a second tag for pushing to the remote registry, replacing "localhost" by my manager node address (where the registry is at)
Tag my locally built image with that tag and docker-compose push it
Deploy the app with docker --context <staging|production> stack deploy my_app
I'm answering myself since I did solve my original problem, but would love to see other DevOps implementations for similar scenarios.
I use minikube with Docker driver on Linux. For a manual workflow I can enable registry addon in minikube, push there my images and refer to them in deployment config file simply as localhost:5000/anything. Then they are pulled to a minikube's environment by its Docker daemon and deployments successfully start in here. As a result I get all the base images saved only on my local device (as I build my images using my local Docker daemon) and minikube's environment gets cluttered only by images that are pulled by its Docker daemon.
Can I implement the same workflow when use Skaffold? By default Skaffold uses minikube's environment for both building images and running containers out of them, and also it duplicates (sometimes even triplicates) my images inside minikube (don't know why).
Skaffold builds directly to Minikube's Docker daemon as an optimization so as to avoid the additional retrieve-and-unpack required when pushing to a registry.
I believe your duplicates are like the following:
$ (eval $(minikube docker-env); docker images node-example)
REPOSITORY TAG IMAGE ID CREATED SIZE
node-example bb9830940d8803b9ad60dfe92d4abcbaf3eb8701c5672c785ee0189178d815bf bb9830940d88 3 days ago 92.9MB
node-example v1.17.1-38-g1c6517887 bb9830940d88 3 days ago 92.9MB
Although these images have different tags, those tags are just pointers to the same Image ID so there is a single image being retained.
Skaffold normally cleans up left-over images from previous runs. So you shouldn't see the minikube daemon's space continuously growing.
An aside: even if those Image IDs were different, an image is made up of multiple layers, and those layers are shared across the images. So Docker's reported image sizes may not actually match the actual disk space consumed.
Hi i have problem with updating / changing image of my service on the server running Docker swarm mode.
Here is the process of manually updating the service.
push the project to gitlab from local machine.
pull the project from gitlab in server.
build a Docker image as my-project:latest
tag my-project:latest as registry.gitlab.com/my-group/my-project:staging
i push the image using docker push registry.gitlab.com/my-group/my-project:staging
i run docker stack deploy -c ~/docker-stack.yml api --with-registry-auth
and it works fine.
However if i move the codes above into a gitlab-ci.yml despite of ending the job successfully i get an error when it is trying to update the service.
Updating service api_backend (id: r4gqmil66kehzf0oehzqk57on)
image registry.gitlab.com/my-group/my-project:staging could not be accessed on a registry to record
its digest. Each node will access registry.gitlab.com/my-group/my-project:staging independently,
possibly leading to different nodes running different
versions of the image.
Also the gitlab runner is executing commands in Shell mode.
I have tried different solutions as you can see i'm even using the --with-registry-auth flag.
To summarize this:
everything works fine if i enter the codes manually but i get an error when i use gitlab-ci.yml.
I am deploying 4 containers using docker stack deploy as below:
docker stack deploy --compose-file compose.yml --with-registry-auth myapp
For the first time, the containers are built using the latest image on the registry, no problem.
But when I push new images to the registry and run the commands again, the containers are not rebuilt using the latest images.
I am using the latest tag in my images. I know it is not the recommended way to do things, but for what I have read in the documentation, docker stack deploy if using the latest tag, will check for image sha with the registry, if it is different the containers will rebuild using latest images, but In my case, it's not happening. Am I missing something here?
I also get an error/warning when I run docker stack deploy once the stack is already up:
Updating service service_name (id: some_hash_value)
image docker.pkg.github.com/username/repository/image-name:latest could not be accessed on a registry to record
its digest. Each node will access docker.pkg.github.com/username/repository/image-name:latest independently,
possibly leading to different nodes running different
versions of the image.
I encoutered the same error message when I started using a new docker registry. The new registry's SSL certificate was not considered secured by docker.
So I got this error until I added my new registry to the insecure-registries section of the /etc/docker/daemon.json
I've seen nobody mentionning this solution on this question or other similar ones, so I hoped this could help.
I've researched around before asking here, but all answers lead me to the same conclusion:
Build your Docker Compose stack locally
Tag and push the images to a registry (either a private or public one like Docker Hub)
Push the stack to the swarm using docker stack deploy --compose-file docker-compose.yml stackdemo
From here, the stack picks up and "pulls" the images from the registry and runs the containers
Is there no straightforward way to make the following (I think common sense) scenario work seamlessly?
Docker swarm manager has access (SSH keys) to pull the project from Git.
It periodically pulls the project and builds it "locally" using docker-compose up
When the build succeeds (containers are ready), it pushes the stack to the swarm using docker stack deploy, propagating the images to all worker nodes.
In that way, the original "source code" is only known by the Manager Node and only it has direct access to the Git repository.
Maintaining a registry (or paying for a cloud one), seems like a huge disadvantage for using Docker in Swarm Mode.
Side note: I've tried the approach of deploying a registry as a service within the stack and tagging + pushing the images to 127.0.0.1/myimage but that led to a different set of problems of its own - e.g. the fact that worker nodes that do not have an instance of the Registry container running, have no access to pull the image (the registry needs to be replicated to all nodes).
Use the docker save and docker load commands to transfer images from your dev machine to all of your swarm machines.
Docker swarm is orchestration & used or intended for managing docker node cluster. When any service is deployed, docker engine can start it on any of the node in the cluster (node that satisfy placement constraints if provided). Now, if one dont have registry, & images are available on node locally, docker cant verify if all nodes will point to same version of the image. Hence, swarm pulls image from registry & then deploys it to nodes.
Having registry also helps in keeping copy of images & registry keeps all version even if images are prune on one or all of docker nodes. One can enable backup of registry & hence there's no chance for loss of any image built & pushed to a registry.
Starting a registry (at least on localhost) is very easy - but that's not what your question.
Coming to your question, you can keep the compose stack file in the same directory where you have Dockerfile & then in the stack compose file you can write service which will get build at the time you deploy the stack:
version: "3.9"
services:
web:
image: 127.0.0.1:5000/stackdemo
build: .
ports:
- "8000:8000"
Here,
build: .
this line builds the image with given name tagging to registry on localhost - but is not pushed which needs to be done manually.
So, in Dockerfile you can write git pull or git clone & then run command to build your code & so on.
Here's a link which provides simple steps to start a registry & build image on the fly while deploying the stack:
https://docs.docker.com/engine/swarm/stack-deploy/
Also, swarm does not works without registry & hence it's not possible to just save & load image & use with swarm orchestration.