i'm trying to set up a rke cluster, the connection to the nodes goes well but when it starts to check etcd health returns:
failed to check etcd health: failed to get /health for host [xx.xxx.x.xxx]: Get "https://xx.xxx.x.xxx:2379/health": remote error: tls: bad certificate
If you are trying to upgrade the RKE and facing this issue then it could be due to the missing of kube_config_<file>.yml file from the local directory when you perform rke up.
This similar kind of issue was reported and reproduced in this git link . Can you refer to the work around and reproduce it by using the steps provided in the link and let me know if this works.
Refer to this latest SO and doc for more information.
Related
Our application is docker based and requires Nat network to be created on the host machine in order to communicate since its a web service. It was working since last 4 months and suddenly stopped working. Checked and found that docker service is stopped. Manually tried restarting the service but it failed to start. Below is the error in the event log:
Error:
fatal: failed to start deamon: Error initializing network controller: Error creating default network: failed during hnsCallRawResponse: hnsCall failed in Win32: There are no more endpoints available from endpoint mapper. (0x6d9)
Tried the below steps:
Deleted the hns.data and restarted the hns service. Then restarted the docker engine service. The issue persists.
Tried running MOFCOMP. Same issue.
Tried removing docker and reinstalling it. Doesn't work.
Tried creating nat network manually. But getting the above mentioned error.
Can someone help here? what needs to be checked or what could be the reason for this issue?
I have a Fabric project up and running with 7 org/5 channel setup with each org having 2 peers. Everything is up and running. Now i am trying to connect Hyperledger Explorer to view the blockchain data. However there is an issue i am facing in the configuration part.
Steps i performed:
Pulled the images and added the following containers in a single docker-compose.yaml file for startup: hyperledger/explorer-db:latest, hyperledger/explorer:latest, prom/prometheus:latest, grafana/grafana:latest
Edited the created containers with the respective configurations needed and volume mounts.
volumes:
./config.json:/opt/explorer/app/platform/fabric/config.json
./connection-profile:/opt/explorer/app/platform/fabric/connection-profile/
./crypto-config:/tmp/crypto
walletstore:/opt/wallet
Since its a multi-org setup i edited the config.json files and accordingly pointed them to the respective connection profiles as per the organization setup
{
"network-configs": {
"org1-network": {
"name": "Sample-1",
"profile": "./connection-profile/org1-network.json"
}, and so on for other orgs
Edited the prometheus.yml to put in the static configurations
static_configs:
targets: ['localhost:8443','localhost:8444', and so on for every peer service]
targets: ['orderer0-service:8443','orderer1-service:8444', and so on for every orderer service]
Edited the peer services in my docker-compose.yaml file to add in the below values on each peer config
CORE_OPERATIONS_LISTENADDRESS=0.0.0.0:9449 # RESTful API for Hyperledger Explorer
CORE_METRICS_PROVIDER=prometheus # Prometheus will pull metrics
Issue: (Now resolved - see below)
It seems that explorer isn't able to find my Admin#org1-cert.pem' path in the given location. But i double checked everything and that particular path is present and also accessible. All permissions to that path is also open to avoid any permissioning issue.
Path in question [Full path is provided not the relative path]: /home/auro/Desktop/HLF/fabricapp/crypto-config/peerOrganizations/org1/users/Admin#org1/msp/signcerts/Admin#org1-cert.pem
The config files is also setup properly. I am unable to find a way to correct way. Would be really glad if someone can tell me what is going on with this path issue, because i tried everything i think i could but still not able to get it working.
Other details:
Using Hypereldger Explorer - v1.1.0 - Pulling the latest docker image
Using Hyperledger Fabric - v.1.4.6 - Pulling the specific version from docker hub for this
Update: Okay, i managed to solve this. Apparently the path to be given in the config file isnt that of the local system but of the docker container. I replaced the path with the path to my docker container where the files are placed and it worked.
New Problem -1: (Now solved) Now i am getting an error as shown below. Highlighted in yellow
I had a look at peer-0-org-1-service node logs when this happened and this is the error it had logged.
2020-07-20 04:38:15.995 UTC [core.comm] ServerHandshake -> ERRO 028 TLS handshake failed with error tls: first record does not look like a TLS handshake server=PeerServer remoteaddress=172.18.0.53:33300
Update: Okay, i managed to solve this too. There were 2 issues. The TLS handshake wasn't happening because the TLS certificate wasn't set to true in the config. The second issue of STREAM removed happened because the url in the config wasnt specified as grpc. Once changes were done, it resolved
New Problem -2: (Current Issue)
It seems that the channel issue is there. Somehow it still shows "not assigned to this channel" and a new error of "Error: 14 UNAVAILABLE: failed to connect to all addresses". This same error happened for all the peers across 7 orgs.
And not to mention suddenly the peers are not able to talk to each other.
Error Received: Could not connect to Endpoint: peer0-org2-service:7051, InternalEndpoint: peer0-org2-service:7051, PKI-ID: , Metadata: : context deadline exceeded
I checked the peer channel connection details and everything seems to be in order. Stuck in this for now. Let me know if anyone has any ideas.
As you can see from the edits i got one problem solved before another came along. After banging my head for a lot of times, i removed the entire build, rebuilt it again with my corrections given above and it simply started working.
You seem to be using old Explorer image. I strongly recommend to use the latest one v1.1.1. Note: There are some updates of settings format in connection profile (e.g. login credential of Explorer). Please refer README-CONFIG for detail.
I installed kubernetes cluster (include one master and two nodes), and status of nodes are ready on master. When I deploy the dashboard and run it by acccessing the link http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/, I get error
'dial tcp 10.32.0.2:8443: connect: connection refused' Trying to
reach: 'https://10.32.0.2:8443/'
The pod state of dashboard is ready, and I tried to ping to 10.32.0.2 (dashboard's ip) not succesfully
I run dashboard as the Web UI (Dashboard) guide suggests.
How can I fix this ?
There are few options here:
Most of the time if there is some kind of connection refused, timeout or similar error it is most likely a configuration problem. If you can't get the Dashboard running then you should try to deploy another application and try to access it. If you fail then it is not a Dashboard issue.
Check if you are using root/sudo.
Have you properly installed flannel or any other network for containers?
Have you checked your API logs? If not, please do so.
Check the description of the dashboard pod (kubectl describe) if there is anything suspicious.
Analogically check the description of service.
What is your cluster version? Check if any updates are required.
Please let me know if any of the above helped.
Start proxy, if it's not started
kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='.*'
On Linux, with minikube v0.34.1, when run minikube start --logtostderr, get following error:
I0227 18:25:12.625477 13250 kubernetes.go:121] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.102:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.102:8443: connect: connection refused]
And, none of following environment variable is set: $HTTP_PROXY, $HTTPS_PROXY, $NO_PROXY.
After searching via google, and checked following posts, still unsolved:
minikube may fail with older VM's and apiserver.Authorization.Mode=RBAC: kube-proxy timeout #2948
HTTP_PROXY set: error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable #2726
Following actions have been tried, with no good news:
minikube delete; minikube start
rm -rf ~minikube/
As a newbie to K8s, really don't understand what this means, any idea ?
#Update - Seems solved
The solution moved to the answer section
I just moved the solution posted by Eric Wang from the question to the answer section:
With following steps seems it's resolved:
Make a backup of ~/.minikube/cache/, optionally.
Otherwise, will need to download those caches again.
Removed config & data via rm -rf ~/.minikube/
mkdir ~/.minikube/
Then restore .minikube/cache/, if you did a backup.
minikube stop
minikube delete
minikube start --logtostderr
Tips:
The --logtostderr flag is useful to get error info on the console.
Without it, the process can stuck there without giving you any information.
I am creating a sample Blockchain network using tutorial https://hyperledger-fabric.readthedocs.io/en/release-1.2/build_network.html. I am facing an error while connecting the peers :
Error: failed to create deliver client: orderer client failed to connect to orderer.example.com:7050: failed to create new connection: context deadline exceeded.
I found a probable solution here which I would like to test but I need help for below :
How to update default network of the containers
How to add property for each container.
While accessing my etc/docker directory I am getting error 'Server returned empty listing for directory '/etc/docker' and also it says permission denied when I try to access it from terminal. Any help will be appreciated.
There is no need for making any changes in Docker containers. I faced similar issue, you can clean up the system space or if you are using a VM you install a fresh network in a new VM(assuming you already have all configuration files copied).