Docker: Swarm worker nodes not finding locally built image - docker

Question

Maybe I missed something, but I made a local docker image. I have a 3 node swarm up and running. Two workers and one manager. I use labels as a constraint. When I launch a service to one of the workers via the constraint it works perfectly if that image is public.
That is, if I do:
docker service create --name redis --network my-network --constraint node.labels.myconstraint==true redis:3.0.7-alpine
Then the redis service is sent to one of the worker nodes and is fully functional. Likewise, if I run my locally built image WITHOUT the constraint, since my manager is also a worker, it gets scheduled to the manager and runs perfectly well. However, when I add the constraint it fails on the worker node, from docker service ps 2l30ib72y65h I see:
... Shutdown Rejected 14 seconds ago "No such image: my-customized-image"
Is there a way to make the workers have access to the local images on the manager node of the swarm? Does it use a specific port that might not be open? If not, what am I supposed to do - run a local repository?

Answer 1

The manager node doesn't share out the local images from itself. You need to spin up a registry server (or user hub.docker.com). The effort needed for that isn't very significant:
# first create a user, updating $user for your environment:
if [ ! -d "auth" ]; then
mkdir -p auth
fi
touch auth/htpasswd
chmod 666 auth/htpasswd
docker run --rm -it \
-v `pwd`/auth:/auth \
--entrypoint htpasswd registry:2 -B /auth/htpasswd $user
chmod 444 auth/htpasswd
# then spin up the registry service listening on port 5000
docker run -d -p 5000:5000 --restart=always --name registry \
-v `pwd`/auth/htpasswd:/auth/htpasswd:ro \
-v `pwd`/registry:/var/lib/registry \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Local Registry" \
-e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
-e "REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry" \
registry:2
# then push your image
docker login localhost:5000
docker tag my-customized-image localhost:5000/my-customized-image
docker push localhost:5000/my-customized-image
# then spin up the service with the new image name
# replace registryhost with ip/hostname of your registry Docker host
docker service create --name custom --network my-network \
--constraint node.labels.myconstraint==true --with-registry-auth \
registryhost:5000/my-customized-image

Answer 2

For me, this step-by-step guide worked. However, it is insecure:
# Start your registry
$ docker run -d -p 5000:5000 --name registry registry:2
# Tag the image so that it points to your registry
$ docker tag my_existing_image localhost:5000/myfirstimage
# Push it to local registry/repo
$ docker push localhost:5000/myfirstimage
# For verification you can use this command:
$ curl -X GET http://localhost:5000/v2/_catalog
# It will print out all images on repo.
# On private registry machine add additional parameters to enable insecure repo:
ExecStart=/usr/bin/dockerd --insecure-registry IP_OF_CURRENT_MACHINE:5000
# Flush changes and restart Docker:
$ systemctl daemon-reload
$ systemctl restart docker.service
# On client machine we should say docker that this private repo is insecure, so create or modifile the file '/etc/docker/daemon.json':
{ "insecure-registries":["hostname:5000"] }
# Restart docker:
$ systemctl restart docker.service
# On swarm mode, you need to point to that registry, so use host name instead, for example: hostname:5000/myfirstimage

Answer 3

Images have to be downloaded to the local cache on each node. The reason is that if you store all of your images on one node only and that node goes down, swarm would have no way to spawn new tasks (containers) on the other nodes.
I personally just pull a copy of all the images on each node before starting the services. That can be done in a bash script or Makefile (eg below)
pull:
#for node in $$NODE_LIST; do
OPTS=$$(docker-machine config $$node)
set -x
docker $$OPTS pull postgres:9.5.2
docker $$OPTS pull elasticsearch:2.3.3
docker $$OPTS pull schickling/beanstalkd
docker $$OPTS pull gliderlabs/logspout
etc ...
set +x
done