I have installed a Kubenetes deployment (version:1.19.14) with docker version 20.10.8 on unbuntu 18.04.
I was able to install it and was working fine.
Due to some reason internet connectivity was lost on the host and on some finding I found that proxy settings were erased.
When I added the proxy the internet connectivity started working but strangely I was not able to give kubernetes kubectl commands anymore.
While trying kubectl commands after exporting proxy, the following error pops up:
Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exeeded while awaiting headers)
We exported the proxy in the following manner:
export http_proxy=http://proxy.example.com:80
export https_proxy=$http_proxy
I searched and was suggested somewhere to make the proxy persistent through http-proxy.conf and reload the daemon:
sudo systemctl daemon-reload
sudo systemctl restart docker
Even after doing this the kubectl commands didn't work.
Please let me know how can I resolve this issue.
Kubectl is just a CLI that communicates with the api-server of the Kubernetes control plane. First of all you need to make sure that the api-server is running and healthy, and that this is not the source of your problem.
You can use the tool crictl to debug pods when Kubectl is not working. It takes directly to the underlying container runtime which would be containerd if you are using Docker.
Related
I'm running a k8 cluster on Docker for Mac. To allow a connection from my database client to my mysql pod, I use the following command kubectl port-forward mysql-0 3306:3306. It works great, however a few hours later I get the following error E0201 18:21:51.012823 51415 portforward.go:233] lost connection to pod.
I check the actual mysql pod, and it still appears to be running. This happens every time I run the port-forward command.
I've seen the following answer here: kubectl port forwarding timeout issue and the solution is to use the following flag --streaming-connection-idle-timeout=0 but the flag is now deprecated.
So following on from there, It appears that I have to set that parameter via a kubelet config file (config file)? I'm unsure on how I could achieve this as Docker for Mac runs as a daemon and I don't manually start the cluster.
Could anyone send me a code example or instructions as to how i could configure kubectl to set that flag so my port forwarding won't have timeouts?
Port forwards are generally for short term debugging, not “hours”. What you probably want is a NodePort type service which you can then connect to directly.
I am going through the kubernetes tutorial at Udacity. When i run the the nginx image using the following command
kubectl run nginx --image=nginx:1.10.0
It given me the error
error: failed to discover supported resources: Get http://localhost:8080/apis/extensions/v1beta1: dial tcp 127.0.0.1:8080: getsockopt: connection refused
If i try to get pods using the following command
kubectl get pods
it says
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The nginx server is running, i can tell because i can get the appropriate output by running curl http://127.0.0.1
I am not able to figure out what the issue is, and there are not a lot of resources on the internet for this problem. Can anyone please tell me how do i resolve it?
This issue often occurs when kubectl can not find the configuration credential for the intended cluster.
Check $HOME/.kube/config for cluster configuration. If configuration is empty or configuration is set for a wrong cluster, regenerate configuration by running,
gcloud container clusters get-credentials <CLUSTER_NAME> --zone <ZONE>
This will update the configuration credential in $HOME/.kube/config.
Now, everything should work as expected.
Reference: https://github.com/googlecodelabs/feedback/issues/537
Check your kubectl config file (~/.kube/config)
For testing purposes, you can use the admin one:
kubectl --kubeconfig /etc/kubernetes/admin.conf get po
Or (again, for testing)
sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf
You can see more suggestions in kubernetes/kubernetes issue 23726
As commented below, that requires kubernetes to be installed, for the node to be able to join a cluster:
sudo kubeadm join --token TOKEN MASTER_IP:6443
The solution was simple, as #VonC suggested, i did not have kubernetes installed, i followed this tutorial, and now i can proceed with my work.
failed to discover supported resources ......
kubectl command line tools connects with kube-apiserver at port 8443 for its operations.
To proble whether the apiserver is up, try curl https://192.168.99.100:8443
If it fails, it means kube-apiserver is not running.
Most probably minikube would not be running.
So try:
minikube status
minikube start
OR
restart the VM
In some cases, it is simply because you need the kubectl run command as root (e.g. sudo it).
You need to set up the zone first:
gcloud config set compute/zone us-central1-b
then add a cluster there :
gcloud container clusters create io
now you can run the commands .
Let me know if found a problem there :)
I am running kubernetes on minikube, I am behind a proxy, so I had set the env variables(HTTP_PROXY & NO_PROXY) for docker in /etc/systemd/system/docker.service.d/http-proxy.conf.
I was able to do docker pull but when I run the below example
kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort
kubectl get pod
pod never starts and I get the error
desc = unable to pull sandbox image \"gcr.io/google_containers/pause-amd64:3.0\"
docker pull gcr.io/google_containers/echoserver:1.4 works fine
I ran into the same problem and am sharing what I learned after making a couple of wrong turns. This is with minikube v0.19.0. If you have an older version you might want to update.
Remember, there are two things we need to accomplish:
Make sure kubctl does not go through the proxy when connecting to minikube on your desktop.
Make sure that the docker daemon in minikube does go through the proxy when it needs to connect to image repositories.
First, make sure your proxy settings are correct in your environment. Here is an example from my .bashrc:
export {http,https,ftp}_proxy=http://${MY_PROXY_HOST}:${MY_PROXY_PORT}
export {HTTP,HTTPS,FTP}_PROXY=${http_proxy}
export no_proxy="localhost,127.0.0.1,localaddress,.your.domain.com,192.168.99.100"
export NO_PROXY=${no_proxy}
A couple things to note:
I set both lower and upper case. Sometimes this matters.
192.168.99.100 is from minikube ip. You can add it after your cluster is started.
OK, so that should take care of kubectl working correctly. Now we have the next issue, which is making sure that the Docker daemon in minikube is configured with your proxy settings. You do this, as mentioned by PMat like this:
$ minikube delete
$ minikube start --docker-env HTTP_PROXY=${http_proxy} --docker-env HTTPS_PROXY=${https_proxy} --docker-env NO_PROXY=192.168.99.0/24
To verify that theses settings have taken, do this:
$ minikube ssh -- systemctl show docker --property=Environment --no-pager
You should see the proxy environment variables listed.
Why do the minikube delete? Because without it the start won't update the Docker environment if you had previously created a cluster (say without the proxy information). Maybe this is why PMat did not have success passing --docker-env to start (or maybe it was on older version of minikube).
I was able to fix it myself.
I had Docker on my host and there is Docker in Minikube.
Docker in Minukube had issues
I had to ssh into minikube VM and follow this post
Cannot download Docker images behind a proxy
and it all works nows,
There should be a better way of doing this, on starting minikube i have passed docker env like below, which did not work
minikube start --docker-env HTTP_PROXY=http://xxxx:8080 --docker-env HTTPS_PROXY=http://xxxx:8080
--docker-env NO_PROXY=localhost,127.0.0.0/8,192.0.0.0/8 --extra-config=kubelet.PodInfraContainerImage=myhub/pause:3.0
I had set the same env variable inside Minikube VM, to make it work
It looks like you need to add the minikube ip to no_proxy:
export NO_PROXY=$no_proxy,$(minikube ip)
see this thread: kubectl behind a proxy
I'm a newbie in docker. I try to run
$ docker run hello-world
And I got this error:
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
Pulling repository docker.io/library/hello-world
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/hello-world/images. You may want to check your internet connection or if you are behind a proxy..
See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.
Also I allowed all points for this app in firewall
screenshot
I ran into that issue while moving from mac to windows. I would recommend to check the following things:
How many network adapters do you have? Check if there are too many and if docker is using the correct one. In case you have too many - delete the one which is not being used
for some people the network protocol IP6 needed to be deactivated to get docker to talk to the internet
check your traffic on the network adapter - in case you are using hyper-v you need to make sure that the network adapter is allowed to use the one which is connected to the internet
I hope this gave you a notch into the right direction.
I resolved this issue on windows 10 by resetting the DNS server to use the Google DNS fixed address: 8.8.8.8 (docker documentation)
After installing the docker for windows, I got the error as below.
PS C:\Users<Username>> docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled (Client.Timeout exceeded while awaiting headers).
So made changes in docker > Settings > Resources > Network > updated docker subnet to
10.0.75.0/24.
It worked like charm after.
When I run kubectl run ... or any command I get an error message saying
The connection to the server localhost:8080 was refused - did you specify the right host or port?
What exactly is this error and how to resolve it?
In my case, working with minikube I had not started minikube. Starting minikube with
minikube start
fixed it.
In most cases, this means a missing kubeconfig file. kubectl is trying to use the default values when there is no $HOME/.kube/config.
You must create or copy a valid config file to solve this problem.
For example if you are using kubeadm you can solve this with:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively you can also export KUBECONFIG variable like this:
export KUBECONFIG=/etc/kubernetes/admin.conf
I really don't know much about kubectl... But the various reasons you have a connection refused to localhost I know of are as follows
1) Make sure you can resolve and ping your local host with the IP(127.XX.XX.XX) and also "localhost" if using a DNS or host file.
2) Make sure the service trying to access the localhost has enough permissions to run as root if trying to use localhost.
3) Check the ports with netstat and check for the appropriate flags you need amongst the "Plantu" flags, Look up the meaning of each of the flags as it would apply to your situation. If the service you are trying to access on localhost is listening on that port, netstat would let you know.
4) Check if you have admin or management settings in your application that needs permissions to access your localhost in the configuration parameters of your application.
5) According to the statement that says did you specify the right host or port, this means that either your "kubectl" run is not configured to run as localhost instead your primary DNS server hostname or IP, Check what host is configured to run your application and like I said check for the appropriate ports to use, You can use telnet to check this port and further troubleshoot form there.
My two cents!
creating cluster before running kubectl worked for me
gcloud container clusters create k0
If swap is not disabled, kubelet service will not start on the masters and nodes, for Platform9 Managed Kubernetes version 3.3 and above..
By running the below command to turn off swap memory
sudo swapoff -a
To make it permanent
go to /etc/fstab and comment the swap line
works well..
I'm a newbie in k8s, came here while working with microk8s &
want to use kubectl on microk8s cluster.
run below command
microk8s config > ~/.kube/config
got the solution from this link
https://microk8s.io/docs/working-with-kubectl
overall, kubectl needs a config file to work with cluster (here microk8s cluster)
Thanks
I also experienced the same issue when I executed kubectl get pods. The reason was docker desktop was not running, then I ran the docker desktop, checked for the docker and k8s running . Then again I ran kubectl get pods
same output. Then I started minikube by minikube start. Everything went normal.
try run with sudo permission mode
sudo kubectl run....