First of all, to mention I checked multiple questions in Stackoverflow but the requirement was somewhat different or the solution offered was not working in my case, so thought of creating a new question.
I am using Local Registry and able to push/pull image from all worker nodes as well
Registry service is up and running in all nodes
Issue while creating service of local image that I already pushed to the docker.
Issue:
overall progress: 0 out of 1 tasks
1/1: ready [======================================> ]
verify: Detected task failure
Steps I have done:
docker service create --name registry --publish 5000:5000 armbuild/registry (mine is raspi so used armbuild)
docker tag XYZImage localhost:5000/XYZImage -> Working Fine
docker push localhost:5000/XYZImage -> Working Fine
docker service create --name XYZService --replicas 2 localhost:5000/XYZImage --> Issue
Note: Even I tried using IP address and adding that address to insecure registries in daemon.json file.
Any leads? or if I am missing something?
Each container or node is writing to its own volume (/var/lib/registry). You must use a distributed storage driver if you want to use replicas.
From the documentation:
The storage back-end you use determines whether you use a fully scaled
service or a service with either only a single node or a node
constraint.
If you use a distributed storage driver, such as Amazon S3, you can
use a fully replicated service. Each worker can write to the storage
back-end without causing write conflicts.
If you use a local bind mount or volume, each worker node writes to
its own storage location, which means that each registry contains a
different data set. You can solve this problem by using a
single-replica service and a node constraint to ensure that only a
single worker is writing to the bind mount.
Related
I have docker images of different elk stacks, and I want to communicate between them. I have achieved it by creating a docker network and accessing them via hostname. I want to know if we can pass this properties in the kubernetes or not?
Can we create a docker network over there? And how do we pass these properties inside the deployment yaml?
I have created a docker network named as "elk", and then passed it in the run arguments (as docker run --network=elk -h elasticsearch ....)
I am expecting to create this network in kubernetes cluster and then pass these properties to deployment yaml
Kubernetes does not have Docker's notion of separate per-application isolated networks. You can't reproduce this Docker setup in Kubernetes and don't need to. Also see Services, Load Balancing, and Networking in the Kubernetes documentation.
In Kubernetes you usually do not communicate directly with Pods (containers). Instead, you also create a Service matching each Deployment, and then make calls to the Service name and port.
If you're currently deploying containers with docker run --net=... then you can ignore that option when migrating to Kubernetes. If you're using Compose, I'd suggest first trying to update the Compose setup to use only the Compose-provided default network, removing all of the networks: blocks.
For something like Elasticsearch, you probably want to run it in a StatefulSet which can also manage the per-replica storage. This has specific requirements around corresponding Services, and it does provide a way to connect to a specific replica when you need to. Relevantly to this question, if the StatefulSet is named elasticsearch then the Pods will be named elasticsearch-0, elasticsearch-1, and so on, and these names will also be visible as the hostname(8) inside the container, matching the docker run -h option.
I have recently started learning docker. However when studying swarm mode I see that containers can be scaled up. What I would like to know is once you scale conatiner in replicated mode will the data within the container be replicated too ? or just fresh containers will be spawned ?
For example lets say I created mysql service initially only with 1 copy. I create and update tables in that mysql container. Later I scale it to 3, will newly spawned containers contain same table data ? Also will the data be continuously be replicated across 3 docker instances ?
A replicated service will use fresh container instances per container. Swarm does not take care about replication of persistent data to be stored in volumes.
Dependening on the volume plugin (e.g. local driver /w remote nfs shares) you are limited to read-write-once or read-write-many. Even if your volume allows read-write-many, the service replicas might not support that, for instance mysql will not work if you point n replicas to the same volume. You can leverage swarm service template variables for instance to point your volumes to different target folders of the same nfs share.
Also with swarm, you will want to have storage that needs to be reachable from all nodes, as a container can die and be re-spawned on a different node. So either you will need to use a remote share based on NFS or CIFS (see example usages nfs cifs), a storage cluster like Ceph or GlusterFS or a cloud native storage like Portworx. While you have to take care of HA for remote share solutions, data replication is build in for storage clusters and cloud native storage.
In case a containerized service itself is cluster/replica aware it is usualy better to not use the swarm replica mechanism - unless all instances can be started with the same set of parameters.
I have setup a 3 node cluster (with no Internet access) with 1 manager and 2 worker-nodes using the standard swarm documentation.
How does the swarm manager in swarm mode know about the images present in worker nodes?
Lets say I have image A in worker-node-1 and image B in worker-node-2 and no images in the manager-node.
Now how do I start container for image A using the manager?
Will it start in manager or node-1?
When I query manager for the list of images will it give the whole list with A and B in it?
Does anyone know how this works?
I couldn’t get the details from the documentation.
Docker Swarm manager node may to be a worker one by the second role but not strictly necessary.
Image deployment policy is mapped via docker-compose.yml which has an information like target nodes, networks, hostnames, volumes, etc. in relation of particular service. So, it will start either in specified node or in emptiest default one.
Swarm manager communicates with the worker nodes via Docker networks:
When you initialize a swarm or join a Docker host to an existing swarm, two new networks are created on that Docker host:
an overlay network called ingress, which handles control and data
traffic related to swarm services. When you create a swarm service and
do not connect it to a user-defined overlay network, it connects to
the ingress network by default
a bridge network called
docker_gwbridge, which connects the individual Docker daemon to the
other daemons participating in the swarm.
Reference
During Swarm deployment, the images of it's services are being propagated to worker nodes according to their deployment policy.
The manager node will contain images once the node is the worker one too (correct me, if it won't).
The default configuration with swarm mode is to pull images from a registry server and use pinning to reference a unique hash for those images. This can be adjusted, but there is no internal mechanism to distribute images within a cluster.
For an offline environment, I'd recommend a stand alone registry server accessible to the cluster. You can even run it on the cluster. Push your image there, and point your server l services to the registry for their images to pull. See this doc for details on running a stand alone registry, or any of the many 3rd party options (e.g. Harbor): https://docs.docker.com/registry/
The other option is to disable the image pinning, and manually copy images to each of your swarm nodes. You need to do this in advance of deploying any service changes. You'll also lose the benefit of reused image layers when you manually copy them. Because of all this issues it creates, overhead to manage, and risk of mistakes, I'd recommend against this option.
Run the docker stack deploy command with --with-registry-auth that will give the Workers access to pull the needed image
By default Docker Swarm will pull the latest image from registry when deploying
When I use 'docker service update' on a peer container in my docker swarm, the peer get's replaced by a new one.
The new one has almost the same name e.g.
old: peer1.org1-223d2d23d23 new: one peer1.org1-345245634ff4
It has access to all files like channel.tx, genesis.block and mychannel.block. in the peer/channel-artifacts map. But the new peer has not joined the channel and no chaincode is installed on it.
I can't join the channel or install chaincode, because for peer1.org1 it already is the case. However if I fetch the oldest channel block I can. But this gives a strange situation I think.
So my question is
How can a peer service in docker swarm still be part of the stack/swarm after an service update or downtime without it being a completely new peer container?
When you upgrade a container in Docker, Docker Swarm or Kubernetes, you are essentially replacing the container (i.e. there is really no concept of an in-place upgrade of the container) with another one which receives the same settings, environment, etc.
When running Docker in standalone mode and using volumes, this is fairly transparent as the new container is deployed on the same host as the prior container and therefore will mount the same volumes, etc.
It seems like you are already mounting some type of volume from shared storage / filesystem in order to access channel.tx, etc.
What you also need to do is actually make sure that you use volumes for the persistent storage used / required by the peer (and orderer, etc for that matter).
On the peer side, the two key attributes in core.yaml are:
peer.fileSystemPath - this defaults to /var/hyperledger/production and is where the ledger, installed chaincodes, etc are kept. The corresponding environment variable is CORE_PEER_FILESYSTEMPATH.
peer.mspConfigPath - where the local MSP info is stored. The corresponding environment variable is CORE_PEER_MSPCONFIGPATH.
You will want to mount those as volumes and given you are using Swarm those volumes will need to be available on a shared storage which is available on all of your Swarm hosts.
I'building a mesosphere infrastructure on AWS instances with 3 master servers (running zookeeper, mesos-master, marathon and haproxy) and N slaves (running mesos-slave and docker).
If I run the same container on different slaves marathon downloads on each slave the same image. I would like to share one single nfs export (say on master1) and mount it on every slave in order to have a unique storage for the images.
Im using Ubuntu on the EC2 instances, so the storage driver used by default is device-mapper. I set up the slaves to mount /var/lib/docker/devicemapper and /var/lib/docker/graph but it ends up with this error: "stale NFS file handle"
What I would like to understand is:
There is a way to do it using a different storage driver?
In any case is the docker daemon doing some look on the files in this directory?
Is my approach wrong or possible leading into "cconcurrency access issues?
Instead of using NFS to expose the backing file system, I think it would be easier to set up docker-registry (with a volume on the master1, so the data is persisted there) and on the other nodes pull images via docker protocol by e.g. docker pull master1:5000/image:latest