I am trying to install Kubernetes on my CentOS machine, when I intialize the cluster, I have the following error.
I specify that I am behind a corporate proxy. I have already configured it for Docker in the directory: /etc/systemd/system/docker.service.d/http-proxy.conf
Docker work fine.
No matter how hard I look, I can't find a solution to this problem.
Thank you for your help.
# kubeadm init
W1006 14:29:38.432071 7560 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": x509: certificate signed by unknown authority
W1006 14:29:38.432147 7560 version.go:103] falling back to the local client version: v1.19.2
W1006 14:29:38.432367 7560 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.2
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[WARNING HTTPProxy]: Connection to "https://192.168.XXX.XXX" uses proxy "http://proxyxxxxx.xxxx.xxx:xxxx/". If that is not intended, adjust your proxy settings
[WARNING HTTPProxyCIDR]: connection to "10.96.0.0/12" uses proxy "http://proxyxxxxx.xxxx.xxx:xxxx/". This may lead to malfunctional cluster setup. Make sure that Pod and Services IP ranges specified correctly as exceptions in proxy configuration
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.19.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.13-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
# kubeadm config images pull
W1006 17:33:41.362395 80605 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": x509: certificate signed by unknown authority
W1006 17:33:41.362454 80605 version.go:103] falling back to the local client version: v1.19.2
W1006 17:33:41.362685 80605 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
failed to pull image "k8s.gcr.io/kube-apiserver:v1.19.2": output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: handshake failure
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher
Maybe root certificates on your machine are outdated - so it does not consider certificate of k8s.gcr.io as valid one. This message x509: certificate signed by unknown authority hints to it.
Try to update them: yum update ca-certificates || yum reinstall ca-certificates
I just did a dig to k8s.gcr.io, and I added the IP given by the request to /etc/hosts.
# dig k8s.gcr.io
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.2 <<>> k8s.gcr.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44303
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;k8s.gcr.io. IN A
;; ANSWER SECTION:
k8s.gcr.io. 21599 IN CNAME googlecode.l.googleusercontent.com.
googlecode.l.googleusercontent.com. 299 IN A 64.233.168.82
;; Query time: 72 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Nov 24 11:45:37 CST 2020
;; MSG SIZE rcvd: 103
# cat /etc/hosts
64.233.168.82 k8s.gcr.io
And now it works!
# kubeadm config images pull
W1124 11:46:41.297352 50730 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.19.4
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.19.4
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.19.4
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.19.4
[config/images] Pulled k8s.gcr.io/pause:3.2
[config/images] Pulled k8s.gcr.io/etcd:3.4.13-0
[config/images] Pulled k8s.gcr.io/coredns:1.7.0
Working also with v1.19.2 - I've got the same error.
It seems to be related to the issue mentioned here (and I think also in here).
I re-install kubeadm on the node and ran the kubeadm init workflow again - it is now working with v1.19.3 and the errors are gone.
All master nodes images are pulled successfully.
Also verified with:
sudo kubeadm config images pull
(*) You can run kubeadm init with --kubernetes-version=X.Y.Z (1.19.3 in our case).
I had the same error. Maybe as others say, it's because of an outdated certificate. I believe it's not required to delete anything.
Simple solution was running one of those two commands, which will reconnect to Container repositories via:
podman login
docker login
Source: podman-login
I had this issue on version version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2" when i tried joining a second control panel.
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:v1.9.3: output: E0923 04:47:51.763983 1598 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"k8s.gcr.io/coredns:v1.9.3\": failed to resolve reference \"k8s.gcr.io/coredns:v1.9.3\": k8s.gcr.io/coredns:v1.9.3: not found" image="k8s.gcr.io/coredns:v1.9.3"
time="2022-09-23T04:47:51Z"...
See #99321 it's now k8s.gcr.io/coredns/coredns:v1.9.3 instead of
k8s.gcr.io/coredns:v1.9.3 and i don't now why
by kluevandrew,
refererence: https://github.com/kubernetes/kubernetes/issues/112131
This worked, am using containerd:
crictl pull k8s.gcr.io/coredns/coredns:v1.9.3
ctr --namespace=k8s.io image tag k8s.gcr.io/coredns/coredns:v1.9.3 k8s.gcr.io/coredns:v1.9.3
docker solution:
docker pull k8s.gcr.io/coredns/coredns:v1.9.3
docker tag k8s.gcr.io/coredns/coredns:v1.9.3 k8s.gcr.io/coredns:v1.9.3
Check imageRepository in kubeadm-config configmap (or your kubeadm config file, if You run something like kubeadm init --config=/tmp/kubeadm-config.yml).
Related
I installed vanilla k8s on a Rocky Linux 8.6 with together with docker.
I created the /etc/docker/daemon.json:
{
"insecure-registries":["rocky-master.mfr.org:5000"],
"exec-opts":["native.cgroupdriver=systemd"],
"storage-driver":"overlay2"
}
But my pod on worker1 says:
Warning Failed 5m26s (x4 over 6m48s) kubelet Failed to pull image "rocky-master.mfr.org:5000/sametime-init:20220712-1935": rpc error: code = Unknown desc = failed to pull and unpack image "rocky-master.mfr.org:5000/sametime-init:20220712-1935": failed to resolve reference "rocky-master.mfr.org:5000/sametime-init:20220712-1935": failed to do request: Head "https://rocky-master.mfr.org:5000/v2/sametime-init/manifests/20220712-1935": http: server gave HTTP response to HTTPS client
Any idea?
I am trying to run fabric network on kubernetes multiple node. The problem occurred while instantiating the chaincode. It works fine on single node Kubernetes cluster but gives following error on chaincode instantiation on multiple nodes.
Error on Peer
2020-12-01 11:15:29.083 UTC [endorser] SimulateProposal -> ERRO 32b
[channel1][10d225ff] failed to invoke chaincode name:"lscc" , error:
Post
http://docker:2375/build?networkmode=host&t=nid1-org1peer1-cc-1.0-bb7b63f343a13a21a9c1a0d74aa7d87a8898eaa0f093e1c77941b4fc795223f3b4: Failed to generate platform-specific docker build: Failed to pull
hyperledger/fabric-ccenv:1.4: API error (500): Get
https://registry-1.docker.io/v2/: net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting
headers)
Error on docker-dind
time="2020-12-01T11:15:29.066402366Z" level=warning msg="Error getting
v2 registry: Get https://registry-1.docker.io/v2/: net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)" time="2020-12-01T11:15:29.066733856Z" level=info
msg="Attempting next endpoint for pull after error: Get
https://registry-1.docker.io/v2/: net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting
headers)"
When the fabric's chaincode is instantiated, it is created as a docker's container in the form of a sandbox.
The container above is created in a docker image called fabric-ccenv, which is defined in the peer's configuration (core.yaml).
When looking at the current log, it seems that the above image cannot be pulled.
It seems that it will be solved by executing the code below and downloading the image to the local repository.
docker pull hyperledger/fabric-ccenv:1.4
In addition, the error appears to be a problem related to the docker's DNS/proxy, and I also had the same issue.
I fixed this by setting 8.8.8.8 as the default DNS
-> docker/Preferences/Proxies
check System proxy
-> System Preferences
Then, add my system DNS server 8.8.8.8
I've started k3d with k3d create && k3d start.
All pods fail to start with the following error:
Warning FailedCreatePodSandBox 14s (x2 over 31s) kubelet,
k3d-k3s-default-server Failed to create pod sandbox: rpc error: code
= Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image
"docker.io/rancher/pause:3.1": failed to pull and unpack image
"docker.io/rancher/pause:3.1": failed to resolve reference
"docker.io/rancher/pause:3.1": failed to do request: Head
https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp:
lookup registry-1.docker.io: Try again
As recommended by a k3d contributor, I've exec'ed into the k3d server container and attempted to pull the image manually:
$ docker exec -it k3d-k3s-default-server sh
/ # ctr image pull docker.io/rancher/pause:3.1
docker.io/rancher/pause:3.1: resolving |--------------------------------------|
elapsed: 4.9 s total: 0.0 B (0.0 B/s)
ctr: failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp: lookup registry-1.docker.io: Try again
In the host environment, docker pull docker.io/rancher/pause:3.1 works just fine.
I've seen a number of people resolve the issue by tweaking various DNS settings. But none described how they arrived at their particular solution.
Solving this issue would make me happy. Discovering a general diagnosis strategy would make me even happier.
What hasn't worked
From here:
I got the issue. I had one entry in
/etc/systemd/network/en0.networking Deleted that file, and everything
is fine.
I have no files in /etc/systemd/network/.
I had the same issue with k3s not being able to pull images and solved it by updating my /etc/resolv.conf to be symlinked from /run/systemd/resolve/stub-resolv.conf on the host machine with
ln -sf /etc/resolv.conf /run/systemd/resolve/stub-resolv.conf
I am trying to install the Kubernetes cluster on ubuntu 18.04 system..
As part of intializing the cluster with below command getting below error
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.14.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.10: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: remote error: tls: access denied
, error: exit status 1
When i tried to run wget https://k8s.gcr.io/v2/ i am getting below error:
Proxy request sent, awaiting response... 401 Unauthorized
Username/Password Authentication Failed.
Can you please let me know what is the issue.
In my case I have
[root#instance-1 ~]# wget https://k8s.gcr.io/v2/
--2019-08-26 12:58:24-- https://k8s.gcr.io/v2/
Resolving k8s.gcr.io (k8s.gcr.io)... 64.233.167.82, 2a00:1450:400c:c06::52
Connecting to k8s.gcr.io (k8s.gcr.io)|64.233.167.82|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authorization failed.
That can be the proxy issue. Inspect [Kubeadm] Failing to pull images, and more concrete [this answer](https://github.com/kubernetes/kubeadm/issues/1201#issuecomment-494686022.
So try to configure your docker to use proxy and I hope your issue would be resolved.
And below is summarized set of commands for both the methods:
Method 1 : Configuring proxy variables in the /etc/sysconfig/docker file
-Add following configuration in /etc/sysconfig/docker file
# cat /etc/sysconfig/docker
HTTP_PROXY="http://USERNAME:PASSWORD#[your.proxy.server]:[port]"
HTTPS_PROXY="https://USERNAME:PASSWORD#[your.proxy.server]:[port]
-Restart docker
# service docker restart
Method 2:
1) Create a drop-in
# mkdir /etc/systemd/system/docker.service.d
2) Create a file with name /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:
[Service]
Environment="HTTP_PROXY=http://user01:password#10.10.10.10:8080/"
Environment="HTTPS_PROXY=https://user01:password#10.10.10.10:8080/"
Environment="NO_PROXY= hostname.example.com,172.10.10.10"
3) reload the systemd daemon
# systemctl daemon-reload
4) restart docker
# systemctl restart docker
5) Verify that the configuration has been loaded:
# systemctl show docker --property Environment
Environment=GOTRACEBACK=crash HTTP_PROXY=http://10.10.10.10:8080/ HTTPS_PROXY=http://10.10.10.10:8080/ NO_PROXY= hostname.example.com,172.10.10.10
I was trying to upload my image to Google Container Registry, but it return some error and I don't know how to troubleshooting.
$> gcloud docker -- push asia.gcr.io/dtapi-1314/web
The push refers to a repository [asia.gcr.io/dtapi-1314/web]
53ccd4e59f47: Retrying in 1 second
32ca8635750d: Retrying in 1 second
e5363ba7dd4d: Retrying in 1 second
d575d439624a: Retrying in 1 second
5c1cba20b78d: Retrying in 1 second
7198e99c156d: Waiting
6ca37046de16: Waiting
b8f2f07b3eab: Waiting
16681562a534: Waiting
92ea1d98cb79: Waiting
97ca462ad9ee: Waiting
unable to decode token response: read tcp 10.0.2.10:54718->74.125.23.82:443: read: connection reset by peer
I checked permission on my Mac.
$> gsutil acl get gs://asia.artifacts.dtapi-1314.appspot.com
It returned a list of correct permission.
I'd tested push on the cloud console, it works.
Does anyone have clue?
Thanks a lot if anyone could help. :)
Other troubleshooting
gcloud auth login
gcloud docker -- login -p $(gcloud auth print-access-token) -u _token https://asia.gcr.io
gsutil acl get gs://asia.artifacts.{%PROJECT_ID}.appspot.com
Add insecure-registry to dockerd startup command.
--insecure-registry asia.gcr.io
Might be the same cause
gcloud docker -- pull google/python
The error was
Error response from daemon: Get https://registry-1.docker.io/v2/google/python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
docker server log
DEBU[0499] Increasing token expiration to: 60 seconds
ERRO[0500] Error trying v2 registry: Get https://registry-1.docker.io/....../python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
ERRO[0500] Attempting next endpoint for pull after error: Get https://registry-1.docker.io/....../python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
DEBU[0500] Skipping v1 endpoint https://index.docker.io because v2 registry was detected
ERRO[0500] Handler for POST /v1.24/images/create returned error: Get https://registry-1.docker.io/....../python/manifests/latest: read tcp 10.0.2.15:37762->52.45.33.149:443: read: connection reset by peer
Environment
MacOS: 10.11.6
Docker Toolbox (on MAC)
Docker 1.12.3 (Git commit: 6b644ec, Built: Wed Oct 26 23:26:11 2016)
The root cause was stupid, but I'd like to update this for anyone who see this question. I found when I attached my computer to company's WIFI. Then It would work (Still some reset). The cable network of my company is mysterious broken to Google Container Registry. The cable network works for all other services (google/youtube/mobile services) we used but broken to Google Container Registry.
Seems like a permission issue. Try running
gcloud auth login
I remember running into a similar issue and this helped.