I have a Hyperledger Fabric network running with Docker swarm. I want to test Fabric by taking some peers down and see if the network still functions.
When the network is running I stop/start a peer container. Then I use the 'docker service update $peer-service --force' command to see if it goes back to being a service. Docker then makes a different new container and adds it to the service.
The new container has not joined the channel and has no chaincode installed on it. The first container still exist but is not part of the swarm anymore. I think it will be very inconvenient to manually install everything on a peer when it goes down on an already running network with many chainscodes.
Is there a way to join the old peer container back as the same service to the swarm?
You need to use volumes so that the block/channel data persists. So map the directory in the peer container that contains this information to a directory on your host machine.
Related
I am trying to set up docker swarm with an overlay network. I have some hosts on aws while others are laptops running Ubuntu(same as on aws). Every node has a static public IP. I have created an overlay network as:
docker network create --driver=overlay --attachable test-net
I have created a swarm network on one of the aws hosts. Every other node is able to join that swarm network.
However when I run docker run -it --name alpine2 --network test-net alpine on any node not on aws, I get the error: docker: Error response from daemon: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded.
But if I run the same on any aws host, then everything is working fine. Is there anything more I need to do in terms of networking/ports If there are some nodes on aws while others are not?
I have opened the ports required for swarm networking on all machines.
EDIT: All the nodes are marked as "active" when listing in the manager node.
UPDATE Solved this issue by opening the respective ports. It now works if all the nodes are Linux based. But when I try to make a swarm with the manager as Linux(ubuntu) os, mac os machines are not able to join the swarm.
check if the node in drain state:
docker node inspect --format {{.Spec.Availability}} node
if yes then update the state:
docker node update --availability active node
here is the explanation:
Resolution
When a node is in drain state, it is expected behavior that you should
not be able to allocate swarm mode resources such as multi-host
overlay network IP addresses to the node.However, swarm mode does not
currently provide a messaging mechanism between the swarm leader where
IP address management occurs back to the worker node that requested
the IP address. So docker run fails with context deadline exceeded.
Internal engineering issue escalation/292 has been opened to provide a
better error message in a future release of the Docker daemon.
source
Check if the below ports are opened on both machines.
TCP port 2377
TCP and UDP port 7946
UDP port 4789
You may use ufw to allow the ports:
ufw allow 2377/tcp
I had a similar issue, managed to fix it by making sure the ENGINE VERSION of the nodes were the same.
sudo docker node ls
Another common cause for this is Ubuntu server installer installing docker using snap, and that package is buggy. Uninstall with snap and install using apt. And reconsider Ubuntu. :-/
In Hyperledger fabric each chaincode deployed runs in a separate docker container.
Hyperledger-composer, therefore, creates a new container at each upgrade of the chaincode. From my understanding composer-rest-server or any other way to interact with the composer channel always relies on the last version that has been deployed.
The framework itself does not stop containers running old chaincodes.
Should I do it manually? Is there a good reason to keep them running?
see Upgrading Hyperledger Fabric Business Network for the answer - you can stop them, yes. I suggest to read the link for more detail
Once an information is written on the Blockchain (via Hyperledger Composer or any other mean), you cannot remove it from the ledger.
Keeping the containers running old chaincodes can be considered as a mean to recover your network (for example, if you made a mistake in the ACL and you cannot access to your network anymore).
You can kill and remove old Docker containers using the following commands:
docker kill ID_OF_THE_OLD_CONTAINER
docker rm ID_OF_THE_OLD_CONTAINER
When I use 'docker service update' on a peer container in my docker swarm, the peer get's replaced by a new one.
The new one has almost the same name e.g.
old: peer1.org1-223d2d23d23 new: one peer1.org1-345245634ff4
It has access to all files like channel.tx, genesis.block and mychannel.block. in the peer/channel-artifacts map. But the new peer has not joined the channel and no chaincode is installed on it.
I can't join the channel or install chaincode, because for peer1.org1 it already is the case. However if I fetch the oldest channel block I can. But this gives a strange situation I think.
So my question is
How can a peer service in docker swarm still be part of the stack/swarm after an service update or downtime without it being a completely new peer container?
When you upgrade a container in Docker, Docker Swarm or Kubernetes, you are essentially replacing the container (i.e. there is really no concept of an in-place upgrade of the container) with another one which receives the same settings, environment, etc.
When running Docker in standalone mode and using volumes, this is fairly transparent as the new container is deployed on the same host as the prior container and therefore will mount the same volumes, etc.
It seems like you are already mounting some type of volume from shared storage / filesystem in order to access channel.tx, etc.
What you also need to do is actually make sure that you use volumes for the persistent storage used / required by the peer (and orderer, etc for that matter).
On the peer side, the two key attributes in core.yaml are:
peer.fileSystemPath - this defaults to /var/hyperledger/production and is where the ledger, installed chaincodes, etc are kept. The corresponding environment variable is CORE_PEER_FILESYSTEMPATH.
peer.mspConfigPath - where the local MSP info is stored. The corresponding environment variable is CORE_PEER_MSPCONFIGPATH.
You will want to mount those as volumes and given you are using Swarm those volumes will need to be available on a shared storage which is available on all of your Swarm hosts.
I have been playing around with Hyperledger to make it run on Kubernetes. And I was successful to do so. The only thing which I was not happy with the solution/work-around for the container that was spun up when chaincode is instantiated by the peer.
Kubernetes is simply not aware of this container as it was not started by Kubernetes and by the peer. And to make the peer and chaincode talk to each other I had to update the docker daemon running on the kubernetes node with dns server ip address of the kube-dns service.
Is it possible to instantiate a chaincode in a way where kubernetes is aware of the container of the chaincode.
And also chaincode container is able to talk to peer in a seamless fashion rather than updating docker daemon process of the node within kubernetes cluster
I have been investigating the same issue you are having. One alternative to using the docker daemon on your kubernetes node is spinning up a new container in your Pod using DnD (Docker in docker) technique. In this way you can successfully instantiate the chaincode container in a natural way (you will be able to use KubeDNS for example) as it will be sharing the same network space as the kubernetes Pod. I couldn't find any tutorial on the internet showing the implementation of this theory but if you find one (or do it yourself) please share it on this thread.
Thank you
Reference:
https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80
I currently have three hosts (docker1, docker2 and docker3) which I have not set up using Docker Machine, each one running the v1.12-rc4 Docker daemon.
I run docker swarm init on docker1, which in turn prints a docker swarm join command which I run on both docker2 and docker3. At that point, running docker info on each host contains the Swarm: active line.
It is at this point that the behavior seems to differ from what I used to get with the standalone Swarm container. Especially, running docker network ls will only show me the networks on the local host, and when trying to create an overlay network, it does not seem like worker nodes are aware of it (i.e. it does not show up on their docker network ls.)
I feel like I have missed out on some important information relating to the workings of the Swarm Mode as opposed to the Swarm container.
What is the correct way of setting up such a cluster without Docker Machine on Docker 1.12 while getting the overlay network feature?
I too thought this was an issue when I first started using it.
This works a little differently in 1.12rc4 - when you deploy a container to your swarm with that network attached to it, it should then create the network on the other nodes as well.
Hope this helps!
Issue
You are using the docker command (used to communicate with your localhost Docker daemon) and not the "swarm" command (used to communicate with the Swarm master).
Solution
It depends on the command you used to start Swarm.
A full step-by-step tutorial (including details on how to deploy an overlay network) is detailled on this answer. I'm sure that reading this will help you ;)
With a network scope of swarm, the network is only propagated to worker nodes on an as-needed basis. If you create a service using that network, and it gets scheduled on that worker node, the network will show up in the docker network ls.
With the now-upcoming 1.13 release, you can get a network that has similar behavior to the non-swarm networks by doing docker network create --attachable .... That network will be valid for both services and normal containers, and will be available to all members of the cluster. As of 1.13.0-rc2, those don't seem to show up in the output of docker network ls.