On Linux, with minikube v0.34.1, when run minikube start --logtostderr, get following error:
I0227 18:25:12.625477 13250 kubernetes.go:121] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.102:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.102:8443: connect: connection refused]
And, none of following environment variable is set: $HTTP_PROXY, $HTTPS_PROXY, $NO_PROXY.
After searching via google, and checked following posts, still unsolved:
minikube may fail with older VM's and apiserver.Authorization.Mode=RBAC: kube-proxy timeout #2948
HTTP_PROXY set: error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable #2726
Following actions have been tried, with no good news:
minikube delete; minikube start
rm -rf ~minikube/
As a newbie to K8s, really don't understand what this means, any idea ?
#Update - Seems solved
The solution moved to the answer section
I just moved the solution posted by Eric Wang from the question to the answer section:
With following steps seems it's resolved:
Make a backup of ~/.minikube/cache/, optionally.
Otherwise, will need to download those caches again.
Removed config & data via rm -rf ~/.minikube/
mkdir ~/.minikube/
Then restore .minikube/cache/, if you did a backup.
minikube stop
minikube delete
minikube start --logtostderr
Tips:
The --logtostderr flag is useful to get error info on the console.
Without it, the process can stuck there without giving you any information.
Related
i'm trying to set up a rke cluster, the connection to the nodes goes well but when it starts to check etcd health returns:
failed to check etcd health: failed to get /health for host [xx.xxx.x.xxx]: Get "https://xx.xxx.x.xxx:2379/health": remote error: tls: bad certificate
If you are trying to upgrade the RKE and facing this issue then it could be due to the missing of kube_config_<file>.yml file from the local directory when you perform rke up.
This similar kind of issue was reported and reproduced in this git link . Can you refer to the work around and reproduce it by using the steps provided in the link and let me know if this works.
Refer to this latest SO and doc for more information.
kubectl -n namespace1 logs -f podname
returns the following error.
Error from server: Get https://ipaddress:10250/containerLogs/namespace1/podname-xxkb9/podname?follow=true: net/http: TLS handshake timeout
Proxies are unset.
unset http_proxy
unset https_proxy
But Still the issue comes.
Could anyone help me with this issue please.
What I know for sure this is not a cert issue. This is API versions mismatching problem or something else related to API. There were a few discussions on stack in the past, I'll attach them in the end. Also, I experienced the same few years ago and at that time I also resolved this problem by kubeadm upgrade
First of all check real error message by running kubectl logs -v9 for maximum verbosity level.
Most probably you checked other commands like kubectl get pods, nodes, etc. None of those commands require the apiserver to contact the kubelet, only kubectl logs does.
And #Kamos asked you absolutely the right question re: exec/attach/portforward. 99% they also doesn't work for you because they also require contacting kubelet directly.
There are a lot of chances you will fix issue with Upgrading kubeadm clusters
References:
1. Kubernetes - net/http: TLS handshake timeout when fetching logs (BareMetal)
2. Kubernetes logs command TLS handshake timeout ANSWER1!!!
3. Kubernetes logs command TLS handshake timeout ANSWER2
4. kubectl logs failed with error: net/http: TLS handshake timeout #71343
Reinstalling Kubernetes without proxy resolved this issue.
I installed kubernetes cluster (include one master and two nodes), and status of nodes are ready on master. When I deploy the dashboard and run it by acccessing the link http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/, I get error
'dial tcp 10.32.0.2:8443: connect: connection refused' Trying to
reach: 'https://10.32.0.2:8443/'
The pod state of dashboard is ready, and I tried to ping to 10.32.0.2 (dashboard's ip) not succesfully
I run dashboard as the Web UI (Dashboard) guide suggests.
How can I fix this ?
There are few options here:
Most of the time if there is some kind of connection refused, timeout or similar error it is most likely a configuration problem. If you can't get the Dashboard running then you should try to deploy another application and try to access it. If you fail then it is not a Dashboard issue.
Check if you are using root/sudo.
Have you properly installed flannel or any other network for containers?
Have you checked your API logs? If not, please do so.
Check the description of the dashboard pod (kubectl describe) if there is anything suspicious.
Analogically check the description of service.
What is your cluster version? Check if any updates are required.
Please let me know if any of the above helped.
Start proxy, if it's not started
kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='.*'
Kubernetes version : v1.6.7
Network plugin : weave
I recently noticed that my entire cluster of 3 nodes went down. Doing my initial level of troubleshooting revealed that /var on all nodes was 100%.
Doing further into the logs revealed the logs to be flooded by kubelet stating
Jan 15 19:09:43 test-master kubelet[1220]: E0115 19:09:43.636001 1220 kuberuntime_gc.go:138] Failed to stop sandbox "fea8c54ca834a339e8fd476e1cfba44ae47188bbbbb7140e550d055a63487211" before removing: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "<TROUBLING_POD>-1545236220-ds0v1_kube-system" network: CNI failed to retrieve network namespace path: Error: No such container: fea8c54ca834a339e8fd476e1cfba44ae47188bbbbb7140e550d055a63487211
Jan 15 19:09:43 test-master kubelet[1220]: E0115 19:09:43.637690 1220 docker_sandbox.go:205] Failed to stop sandbox "fea94c9f46923806c177e4a158ffe3494fe17638198f30498a024c3e8237f648": Error response from daemon: {"message":"No such container: fea94c9f46923806c177e4a158ffe3494fe17638198f30498a024c3e8237f648"}
The <TROUBLING_POD>-1545236220-ds0v1 was being initiated due to a cronjob and due to some misconfigurations, there were errors occurring during the running of those pods and more pods were being spun up.
So I deleted all the jobs and their related pods. So I had a cluster that had no jobs/pods running related to my cronjob and still see the same ERROR messages flooding the logs.
I did :
1) Restart docker and kubelet on all nodes.
2) Restart the entire control plane
and also
3) Reboot all nodes.
But still the logs are being flooded with the same error messages even though no such pods are even being spun up.
So I dont know how can I stop kubelet from throwing out the errors.
Is there a way for me to reset the network plugin I am using ? Or do something else ?
Check if the pod directory exists under /var/lib/kubelet
You're on a very old version of Kubernetes, upgrading will fix this issue.
I am totally new to Cassandra and met the following error when using cqlsh:
cqlsh
Connection error: Could not connect to localhost:9160
I read the solutions from the following link and tried them all. But none of them works for me.
How to connect Cassandra to localhost using cqlsh?
I am working on CentOS6.5 and installed Cassandra2.0 using yum intall dsc20.
I ran into the same issue running the same OS and same install method. While the cassandra service claims that it's starting ok, if you run service cassandra status it would tell me that the process was dead. Here are the steps I took to fix it:
Viewing the log file at /var/log/cassandra/cassandra.log gave told me that my heap size was too small. Manually set the heap size in /etc/cassandra/conf/cassandra-env.sh:
MAX_HEAP_SIZE="1G"
HEAP_NEWSIZE="256M"
Tips on setting the heap size for your system can be found here
Next, the error log claimed the stack size was too small. Once again in /etc/cassandra/conf/cassandra-env.sh find a line that looks like JVM_OPTS="$JVM_OPTS -Xss128k" and raise that number to JVM_OPTS="$JVM_OPTS -Xss256k"
Lastly, the log complained that the local url was misformed and threw a java exception. I found the answer to the last part here. Basically, you want to manually bind your server's hostname in your /etc/hosts file.
127.0.0.1 localhost localhost.localdomain server1.example.com
Hope this helps~
Change:
/etc/cassandra/cassandra.yaml
Whether to start the thrift rpc server.
start_rpc: false
to
start_rpc: true