I have been playing around with Hyperledger to make it run on Kubernetes. And I was successful to do so. The only thing which I was not happy with the solution/work-around for the container that was spun up when chaincode is instantiated by the peer.
Kubernetes is simply not aware of this container as it was not started by Kubernetes and by the peer. And to make the peer and chaincode talk to each other I had to update the docker daemon running on the kubernetes node with dns server ip address of the kube-dns service.
Is it possible to instantiate a chaincode in a way where kubernetes is aware of the container of the chaincode.
And also chaincode container is able to talk to peer in a seamless fashion rather than updating docker daemon process of the node within kubernetes cluster
I have been investigating the same issue you are having. One alternative to using the docker daemon on your kubernetes node is spinning up a new container in your Pod using DnD (Docker in docker) technique. In this way you can successfully instantiate the chaincode container in a natural way (you will be able to use KubeDNS for example) as it will be sharing the same network space as the kubernetes Pod. I couldn't find any tutorial on the internet showing the implementation of this theory but if you find one (or do it yourself) please share it on this thread.
Thank you
Reference:
https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80
Related
The setup below is all done on Google Cloud.
I have a Docker setup running on a VM instance having a Hyperledger Fabric Network of 3 Orderers and 2 peers of a single organization.
I have a Kubernetes Cluster running on GKE having just 2 peer pods.
What I want to achieve is:
I want the peer pod of Kubernetes to be able to communicate with the network setup in Docker.
i.e. I want at least 1 peer of the Kubernetes to be able to join the network channel on the Docker
Network and be able to perform some transaction/query.
Till now what I have done is exposed the services of the 2 peer pods in Kubernetes i.e. ClusterIP -> NodePort.
I have done some reading as to what options are there such as having an Nginx proxy and other things, but I am new to this and still in a beginner phase so a few things are still confusing to me.
Any help would be appreciated.
Since you have two separate container setups (and two different container technologies), containers in one space have to communicate with the other as though they're outside the cluster. You can't join arbitrary Docker containers on another instance into the Kubernetes network environment, nor can you join Kubernetes pods into a single-host Docker network environment.
Using a NodePort or LoadBalancer-type Service as you already have is close to the best you can do, particularly if the underlying protocol isn't HTTP. If you need a multi-host setup, migrating the whole thing into GKE might be a better long-term answer.
I am currently working with a Hyperledger Fabric v1.4 deployment over k8s. The chaincode containers that are generated are basically create by the container running within the peer pods and k8s as such has no knowledge and control of the chaincode containers. In such a scenario where there is a Docker container running along with k8s and k8s has no knowledge of a particular docker container, is it possible for the Docker container to in some way gain access to the k8s master API and gain access to the whole k8s cluster consequently?
My intention with asking this question is to figure out if there is a way to use an container external to any pods in k8s, to cause any undesirable impact to the k8s cluster by gaining unauthorized access to k8s. The chaincode container that I talked about, is created using a trusted template image and the only possible malicious component in the container is a single golang, java or nodejs script that is provided by the user. So my real question here is, "Is it possible using these user scripts gain unauthorized access to the k8s cluster?" And I am primarily focusing on a manager k8s service like Azure Kubernetes Service.
Your question totally changed the meaning so I'll try to rewrite the answer.
You have to remember that the pod you are running the code on by default is limited to just the namespace it's running on. If you didn't gave it any higher privileges. Also the code is not running as root.
You can read about Pod Security Policies and Configure a Security Context for a Pod or Container.
TLDR.
As long as you don't give it any special privileges or rights it should be fairly save for your cluster.
I have a Hyperledger Fabric network running with Docker swarm. I want to test Fabric by taking some peers down and see if the network still functions.
When the network is running I stop/start a peer container. Then I use the 'docker service update $peer-service --force' command to see if it goes back to being a service. Docker then makes a different new container and adds it to the service.
The new container has not joined the channel and has no chaincode installed on it. The first container still exist but is not part of the swarm anymore. I think it will be very inconvenient to manually install everything on a peer when it goes down on an already running network with many chainscodes.
Is there a way to join the old peer container back as the same service to the swarm?
You need to use volumes so that the block/channel data persists. So map the directory in the peer container that contains this information to a directory on your host machine.
I have a kubernetes cluster that is running on AWS EC2 instances and weave as networking(cni). I have disabled the docker networking(ipmask and iptables) as it is managed by weave(to avoid network conflicts).
I have deployed my Jenkins on this cluster as K8s pod and this jenkins uses jenkins kubernetes plugin to spawn dynamic slaves based on pod and container template which I have defined. These slaves container have docker client in it which connects to the host docker engine via docker.sock
So when I run any job in Jenkins it starts a slave and on this it clones a git repo and starts building the Dockerfile present inside the repo.
My sample dockerfile looks like this:
FROM abc:123
RUN yum update
So when container starts building this it tries connecting to redhat repo to update the local repo and fails here. To debug I logged in to this container and try wget/CURL some packages and finds that there is no internet connectivity in this container.
I suspect that while building docker starts intermediate containers and those containers are not managed by weave so they do not have internet connectivity.
Need suggestions.
Related question: Internet connection inside Docker container in Kubernetes
Ok finally after lot of struggle I find the solution.
So when ever K8s starts a pod it starts a sidecart container whose role is basically to provide network to pod containers.
So while running docker build if I pass it's container ID as network then my intermediate contexts start getting internet connectivity via this container.
So changes looks something like this:
docker build -t "some name" --network container:\$(docker ps | grep \$(hostname) | grep k8s_POD | cut -d\" \" -f1) -f infra/docker/Dockerfile .
Hope this helps. :D
You can try to attach weave networking dynamically as a part of your build job. Is it definitely possible to change active network of container on the flight with weave.
Maybe you will need to use some additional container with Weave Docker Api Proxy or you can use a different way to communicate with Weave network on your nodes.
So, the main idea is just attach your containers where you running builds to the Kubernetes pods network, where you have an external access.
Also, and maybe it will be better, you can create another one Weave virtual network with access to the Internet and attach your contenders to it.
You're right - the docker build process runs in a different context, and Weave Net doesn't attach those automatically.
Even more complicated, Kubernetes will connect via CNI whereas Docker has its own plugin API. I believe it's possible to have both on a machine at the same time, but rather complicated.
Maybe look at some of the ways to build images without using Docker ?
In the official Kubernetes multinode Docker guide , it is mentioned that you need to another Docker instance:
A bootstrap Docker instance which is used to start etcd and flanneld, on which the Kubernetes components depend
So what is a bootstrap instance and how do you make sure that keeps running on restarts ?
The documentation gives a detailed explanation as to the purpose of the bootstrap instance of Docker:
This guide uses a pattern of running two instances of the Docker
daemon: 1) A bootstrap Docker instance which is used to start etcd and
flanneld, on which the Kubernetes components depend 2) A main Docker
instance which is used for the Kubernetes infrastructure and user’s
scheduled containers
This pattern is necessary because the flannel daemon is responsible
for setting up and managing the network that interconnects all of the
Docker containers created by Kubernetes. To achieve this, it must run
outside of the main Docker daemon. However, it is still useful to use
containers for deployment and management, so we create a simpler
bootstrap daemon to achieve this.
In summary the special bootstrap docker daemon runs the bits that kubernetes depends on, freeing up the the normal docker daemon to be managed by kubernetes. This is a trick that leverages the fact that both etcd and flanneld can be run as containers. Alternatively one would have to set them up locally as services.
As for ensuring the bootstrapping docker daemon survives a restart, the answer lies within the code. Here's where it's being called when running the master.sh script.
https://github.com/kubernetes/kube-deploy/blob/master/docker-multinode/master.sh#L36
https://github.com/kubernetes/kube-deploy/blob/master/docker-multinode/docker-bootstrap.sh#L20
So the code attempts to setup a service for the extra docker daemon process.