I am using docker swarm and would like to deploy a service with docker-compose. My service uses a custom image called myuser/myrepo:mytag that I successfully deploy to Docker-Hub to a private repository.
My docker-compose looks like this:
version: "3.3"
services:
myservice:
image: myuser/myrepo:mytag
ports:
- "8080:8080"
Before executing, I successfully pulled the image with: docker pull myuser/myrepo:mytag
When I run docker stack deploy -c docker-compose.yml myapp I always receive the error: "No such image: myuser/myrepo:mytag".
Interestingly, running the same file using only: docker-compose up (i.e. without swarm mode) everything works fine and the service starts up.
I really don't understand why this is failing?
I've already tried cleaning up docker with docker system prune and then repull my image, no success.
Already found the solution.
My image is hosted on a private repository.
Besides the swarm manager (where I executed the commands), I had a running swarm worker.
When I ran docker stack deploy -c docker-compose.yml myapp docker deployed the service to the worker node (not the manager node as I thought).
At the worker node, docker had no credentials to pull the image from the private repository.
Hence, to fix this either pass the flag --with-registry-auth (which pushes the credentials for the repository to the worker node) or make sure that the service is deployed to a node where the image is present.
See: https://docs.docker.com/engine/reference/commandline/deploy/
I want to add another scenario that leads to the same outcome (error message) so that people won't bang their heads against the wall.
Another possibility is that you are trying to deploy the image with the insecure registry but forget to edit daemon.json on the server pulling the image.
If that is the case, lets this answer act as a reminder; and save you some time.
I had similar issue on mac when behind the corporate firewall.
I was able to resolve only after connecting directly to internet.
Just to update, while I am on VPN, I am able to access the internet without any proxy settings, and am able to download (docker) images just fine with docker run. Issue is only with docker-compose.
I did try changing the nameserver to 8.8.8.8 in resolv.conf in my VMs, but issue was not resolved.
For me I struggled with an image I had deployed to a new registry I configured in my swarm. I was updating the stack using Portainer.
I configured all the necessary certificates and logins on all the nodes and verified I had uploaded the image using the following commands:
curl -X GET https://myregistry:5000/v2/_catalog
curl -X GET https://myregistry:5000/v2/{image}/tags/list
No matter what I tried I always had the "No such image" error displayed on the service instances.
In a last ditch attempt I created a service (without the compose file) using exactly the same URL for my image as I had previously and it worked, i.e. docker found the image and started the service! Further attempts using the compose file then worked properly for this and all other new images.
Weird.
Related
Hi i have problem with updating / changing image of my service on the server running Docker swarm mode.
Here is the process of manually updating the service.
push the project to gitlab from local machine.
pull the project from gitlab in server.
build a Docker image as my-project:latest
tag my-project:latest as registry.gitlab.com/my-group/my-project:staging
i push the image using docker push registry.gitlab.com/my-group/my-project:staging
i run docker stack deploy -c ~/docker-stack.yml api --with-registry-auth
and it works fine.
However if i move the codes above into a gitlab-ci.yml despite of ending the job successfully i get an error when it is trying to update the service.
Updating service api_backend (id: r4gqmil66kehzf0oehzqk57on)
image registry.gitlab.com/my-group/my-project:staging could not be accessed on a registry to record
its digest. Each node will access registry.gitlab.com/my-group/my-project:staging independently,
possibly leading to different nodes running different
versions of the image.
Also the gitlab runner is executing commands in Shell mode.
I have tried different solutions as you can see i'm even using the --with-registry-auth flag.
To summarize this:
everything works fine if i enter the codes manually but i get an error when i use gitlab-ci.yml.
I am deploying 4 containers using docker stack deploy as below:
docker stack deploy --compose-file compose.yml --with-registry-auth myapp
For the first time, the containers are built using the latest image on the registry, no problem.
But when I push new images to the registry and run the commands again, the containers are not rebuilt using the latest images.
I am using the latest tag in my images. I know it is not the recommended way to do things, but for what I have read in the documentation, docker stack deploy if using the latest tag, will check for image sha with the registry, if it is different the containers will rebuild using latest images, but In my case, it's not happening. Am I missing something here?
I also get an error/warning when I run docker stack deploy once the stack is already up:
Updating service service_name (id: some_hash_value)
image docker.pkg.github.com/username/repository/image-name:latest could not be accessed on a registry to record
its digest. Each node will access docker.pkg.github.com/username/repository/image-name:latest independently,
possibly leading to different nodes running different
versions of the image.
I encoutered the same error message when I started using a new docker registry. The new registry's SSL certificate was not considered secured by docker.
So I got this error until I added my new registry to the insecure-registries section of the /etc/docker/daemon.json
I've seen nobody mentionning this solution on this question or other similar ones, so I hoped this could help.
I've researched around before asking here, but all answers lead me to the same conclusion:
Build your Docker Compose stack locally
Tag and push the images to a registry (either a private or public one like Docker Hub)
Push the stack to the swarm using docker stack deploy --compose-file docker-compose.yml stackdemo
From here, the stack picks up and "pulls" the images from the registry and runs the containers
Is there no straightforward way to make the following (I think common sense) scenario work seamlessly?
Docker swarm manager has access (SSH keys) to pull the project from Git.
It periodically pulls the project and builds it "locally" using docker-compose up
When the build succeeds (containers are ready), it pushes the stack to the swarm using docker stack deploy, propagating the images to all worker nodes.
In that way, the original "source code" is only known by the Manager Node and only it has direct access to the Git repository.
Maintaining a registry (or paying for a cloud one), seems like a huge disadvantage for using Docker in Swarm Mode.
Side note: I've tried the approach of deploying a registry as a service within the stack and tagging + pushing the images to 127.0.0.1/myimage but that led to a different set of problems of its own - e.g. the fact that worker nodes that do not have an instance of the Registry container running, have no access to pull the image (the registry needs to be replicated to all nodes).
Use the docker save and docker load commands to transfer images from your dev machine to all of your swarm machines.
Docker swarm is orchestration & used or intended for managing docker node cluster. When any service is deployed, docker engine can start it on any of the node in the cluster (node that satisfy placement constraints if provided). Now, if one dont have registry, & images are available on node locally, docker cant verify if all nodes will point to same version of the image. Hence, swarm pulls image from registry & then deploys it to nodes.
Having registry also helps in keeping copy of images & registry keeps all version even if images are prune on one or all of docker nodes. One can enable backup of registry & hence there's no chance for loss of any image built & pushed to a registry.
Starting a registry (at least on localhost) is very easy - but that's not what your question.
Coming to your question, you can keep the compose stack file in the same directory where you have Dockerfile & then in the stack compose file you can write service which will get build at the time you deploy the stack:
version: "3.9"
services:
web:
image: 127.0.0.1:5000/stackdemo
build: .
ports:
- "8000:8000"
Here,
build: .
this line builds the image with given name tagging to registry on localhost - but is not pushed which needs to be done manually.
So, in Dockerfile you can write git pull or git clone & then run command to build your code & so on.
Here's a link which provides simple steps to start a registry & build image on the fly while deploying the stack:
https://docs.docker.com/engine/swarm/stack-deploy/
Also, swarm does not works without registry & hence it's not possible to just save & load image & use with swarm orchestration.
I am currently trying to deploy a basic task queue and frontend using celery, rabbitmq and flower on Kubernetes (and minikube). I am following the example here:
https://github.com/kubernetes/kubernetes/tree/release-1.3/examples/celery-rabbitmq
I can get everything to work following the instructions; however, when I run docker build on the Dockerfile in ./celery-app-add, push the image to my own repository and replace endocode/celery-app-add with <mine>/celery-app-add, I can't get the example to run anymore. I am assuming that the Dockerfile in source control is wrong because if I pull the endocode/celery-app-add image and run bash in the image, it loads in as the root user (as opposed to user with <mine>/celery-app-add Dockerfile).
After booting up all of the containers and services, I can see the following in the logs:
2016-08-18T21:05:44.846591547Z AttributeError: 'ChannelPromise' object has no attribute '__value__'
The celery logs show:
2016-08-19T01:38:49.933659218Z [2016-08-19 01:38:49,933: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**#rabbit:5672//: [Errno -2] Name or service not known.
If I echo RABBITMQ_SERVICE_SERVICE_HOST within the container, it appears as the same host as indicated in the rabbitmq-service after running kubectl get services.
I am not really sure where to go from here. Any suggestions are appreciated. Also, I added USER root (won't run this in production, don't worry) to my Dockerfile and still ran into the same issues above. docker history endocode/celery-app-add hasn't been too helpful either.
Turns out the problem is based around this celery issue. Celery prefers to use CELERY_BROKER_URL over anything that can be set in the app configuration. To fix this, I unset CELERY_BROKER_URL in the Dockerfile and it picked up my configuration correctly.
When pushing to the official registry, I get the following error:
Failed to generate layer archive: Error mounting '/dev/mapper/docker-202:1-399203-ed78b67d527d993117331d27627fd622ffb874dc2b439037fb120a45cd3cb9be' on '/var/lib/docker/devicemapper/mnt/ed78b67d527d993117331d27627fd622ffb874dc2b439037fb120a45cd3cb9be': device or resource busy
The first time I tried to push the image, I ran out of memory on my hard drive. After that I cleaned up and should have now enough space to push it, but the first try somehow locked the image. How can I free it again?
I have stopped and removed the container running the image, but that didn't help.
I have restarted the docker service, without any results
This looks like it might be related to the issue mentioned here: https://github.com/dotcloud/docker/issues/4767
It sounds like you've tried stopping and removing the container. Have you tried restarting the docker daemon and/or restarting the host?
I changed docker-compose.yml volume section and gave mysql volumes new name like:
volumes:
- ${DATA_PATH_HOST}/mysql:/var/lib/mysql_new
- ${MYSQL_ENTRYPOINT_INITDB}:/docker-entrypoint-initdb.d
then start docker container and everything works fine.
docker-compose up -d --build mysql