I am running Kubernetes (Minikube) on my local Mac.
I am trying to setup a deployment with Docker image and getting the below error. But, the hello-world deployment with the Docker image "gcr.io/google-samples/node-hello:1.0" works as expected.
I am able to pull the image from a console on my local machine. Am I missing any setting here?
"Failed to pull image
"docker.XYZ.com/dpace/dev/docker-service": rpc error:
code = Unknown desc = Error response from daemon: Get
https:/docker.XYZ.com/v2/: dial tcp: lookup
docker.XYZ.com on 10.0.2.3:53: read udp
10.0.2.15:59292->10.0.2.3:53: i/o timeout"
I am able to pull the image using docker pull docker.XYZ.com/dpace/dev/docker-service in my local machine without any auth issue. It doesn't need auth for pulling images.
I tried logging into Minikube VM and Docker images returns the following.
$ docker images REPOSITORY TAG
IMAGE ID CREATED SIZE
k8s.gcr.io/kubernetes-dashboard-amd64 v1.8.1
e94d2f21bc0c 3 months ago 121MB
gcr.io/google-containers/kube-addon-manager v6.5
d166ffa9201a 4 months ago 79.5MB
gcr.io/k8s-minikube/storage-provisioner v1.8.0
4689081edb10 4 months ago 80.8MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1
4689081edb10 4 months ago 80.8MB
k8s.gcr.io/k8s-dns-sidecar-amd64 1.14.5
fed89e8b4248 5 months ago 41.8MB
k8s.gcr.io/k8s-dns-kube-dns-amd64 1.14.5
512cd7425a73 5 months ago 49.4MB
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64 1.14.5
459944ce8cc4 5 months ago 41.4MB k8s.gcr.io/echoserver
1.4 a90209bb39e3 21 months ago 140MB gcr.io/google_containers/pause-amd64 3.0
99e59f495ffa 22 months ago 747kB k8s.gcr.io/pause-amd64
3.0 99e59f495ffa 22 months ago 747kB gcr.io/google-samples/node-hello 1.0
4c7ea8709739 23 months ago 644MB
Though the images are there, when I try to pull the existing image, it fails with the below error.
$ docker pull gcr.io/google-samples/node-hello:1.0 Error response from
daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on
10.0.2.3:53: read udp 10.0.2.15:44023->10.0.2.3:53: i/o timeout
When I try "docker login docker.XYZ.com", it prompts me to enter the credential. It throws the below error after entering the password. Same error while trying to pull the image also.
"Error response from daemon: Get https://docker.XYZ.com/v2/: dial tcp:
lookup docker.XYZ.com on 10.0.2.3:53: read udp
10.0.2.15:41849->10.0.2.3:53: i/o timeout"
The command "curl google.com" also not working. "Could not resolve
host: google.com"
Any setting to be done inside Minikube VM. I use VirtualBox.
Looks like DNS in your minikube is broken, that's why you cannot pull anything.
Here is an Issue on Github with the similar problem.
Try to update your minikube and your hypervisor (in most of cases it is Virtualbox) to the last version (check here) and recreate a cluster, it should help.
Related
I installed awx-operator on my local server (let's call it GFX server) by following this tutorial: https://github.com/ansible/awx-operator
My network is like:
The GFX server :
eth0: 192.168.1.40 (local address)
tun0: 172.17.3.114 (a vpn address using openvpn)
I have a host BOX1 with:
eth0: 192.168.1.150 (local address)
I have another host BOX2 (not locally)
tun0: 172.17.3.30
When I launch a template with hello_word example on the local host (BOX1) everything went good:
When I try to launch the same template on the distant box (BOX2) I get this error:
"Failed to connect to the host via ssh: ssh: connect to host 172.17.3.30 port 22: No route to host"
Here is a diagram explaining what I have.
here is the output of docker ps command:
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c90bb11eecf5 gcr.io/k8s-minikube/kicbase:v0.0.30 "/usr/local/bin/entr…" 21 hours ago Up 2 hours 127.0.0.1:49157->22/tcp, 127.0.0.1:49156->2376/tcp, 127.0.0.1:49155->5000/tcp, 127.0.0.1:49154->8443/tcp, 127.0.0.1:49153->32443/tcp minikube
e4fdd5c95a5e hawkbit/hawkbit-update-server:latest-mysql "java -jar hawkbit-u…" 12 months ago Up 3 hours 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp hawkbit-server_hawkbit_1
8ab8f6efdcbf mysql:5.7 "docker-entrypoint.s…" 12 months ago Up 3 hours 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp hawkbit-server_mysql_1
Question: there is a solution to let minikube add the route of the VPN ?
Thank you
When i run docker save nifi > nifi_backup.tar i get
Error response from daemon: No such image: nifi
When i run docker ps -a i got
$docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a8796cbf6cb1 apache/nifi:1.9.2 "../scripts/start.sh" 11 months ago Exited (0) 33 minutes ago nifi
f4ecf6ca0d16 apache/nifi "../scripts/start.sh" 12 months ago Exited (0) 12 months ago nervous_poincare
9a68c235bb3a apache/nifi "../scripts/start.sh" 12 months ago Exited (0) 12 months ago jolly_mendel
d608287fe560 superset_superset "/entrypoint.sh" 2 years ago Up 27 hours (healthy) 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp superset_superset_1
fcea620b1983 postgres:10 "docker-entrypoint.s…" 2 years ago Up 27 hours 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp superset_postgres_1
380782e0a024 redis:3.2 "docker-entrypoint.s…" 2 years ago Up 27 hours 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp superset_redis_1
Also, i cant start nifi image
if i run docker start nifi
it try to start about a minute, but then stop silently.
And there is no logs in result.
docker events says exit code = 0:
Maybe it has some relation for unavailable saving, so i inform about it too.
How to fix No such image in such a case?
Use the command docker images to see if there is an image named nifi. If not first build that image and than execute the command:
docker save nifi > nifi_backup.tar
It should not show the error No such image: nifi if the command docker images has an image named nifi.
As per documentation here, docker save command expects an image name, not a container. You're probably looking for docker export (export docs) command to achieve what you want.
The error about starting your container could be container-specific. As it exited already a year ago, maybe data is lost and the container cannot be started again. Using command docker events&, you can start the docker event listener in the background. This way you can obtain the hex value of this very start attempt and use it to search specific logs: docker logs <startId hex>. Maybe it can tell you more details about why the container did not properly start.
I have to install openshift on disconnected system so i followed following steps(original installation requires more image but for sake of understanding i am provided minimum steps)
on system with internet i did following steps
docker pull docker.io/openshift/origin-node:v3.11.0
docker save -o openshift-origin-v3.11.0-images.tar \
docker.io/openshift/origin-node:v3.11.0
on second disconnected system i did following
docker load -i openshift-origin-v3.11.0-images.tar
Now when i start script for installation it pull the images with command docker.io/openshift/origin-node:v3.11.0
which is throwing following error
Error getting v2 registry: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: dial udp [::1]:53: connect: no route to host
When on second system i do docker images
[root#x ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/openshift/origin-node v3.11.0 14d965ab72d5 4 days ago 1.17 GB
Its showing me that image is available. Whats wrong here? My understanding is it should first look locally and then will check from dockerhub
Update1:
if i directly pull it saying
[root#x ~]# docker pull docker.io/openshift/origin-node:v3.11.0
Trying to pull repository docker.io/openshift/origin-node ...
Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.x.x:53: server misbehaving
I am expecting it should say
Status: Image is up to date for
I changed following in /etc/containers/registries.conf and it works
From
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
To
[registries.search]
registries = []
[registries.block]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
I am following a tutorial on the Hyperledger fabric site and after installing all the perquisites (latest versions) on a Linux 18.04 installation I run into an error.
I am trying to run the given ./byfn script to "Build Your First Network". After a fresh install I run the commands as follows:
./byfn generate
./byfn up
At which point everything performs as expected untill the following error occurs 5 times in a row (after which the run exits with an Error):
+ peer channel join -b mychannel.block
+ res=1
+ set +x
Error: error getting endorser client for channel: endorser client failed to connect to peer0.org1.example.com:7051: failed to create new connection: context deadline exceeded
peer0.org1 failed to join the channel, Retry after 3 seconds
I have tried various things like:
Increasing the timeout to allow for longer connection times
I have down-ed the network and upped it again
Full re-installations of required packages and the fabric-samples
Removed all docker volumes/images/containers
I came across some sources mentioning that it might have to do with the peers not being able to connect to each other. Which I tried to fix with a manual docker connect of each peer to the byfn docker network, no success there. I can see the orderer running but the peers that attempted to join the network exited with an error:
docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
99570e191542 hyperledger/fabric-tools:latest "/bin/bash" 24 seconds ago Up 23 seconds cli
340d1225a913 hyperledger/fabric-peer:latest "peer node start" 30 seconds ago Exited (2) 24 seconds ago peer0.org1.example.com
fabe017751a0 hyperledger/fabric-peer:latest "peer node start" 30 seconds ago Exited (2) 25 seconds ago peer1.org2.example.com
f81a639f29f6 hyperledger/fabric-peer:latest "peer node start" 30 seconds ago Exited (2) 26 seconds ago peer1.org1.example.com
0f91080db681 hyperledger/fabric-peer:latest "peer node start" 30 seconds ago Exited (2) 27 seconds ago peer0.org2.example.com
c491adc91320 hyperledger/fabric-orderer:latest "orderer" 30 seconds ago Up 28 seconds 0.0.0.0:7050->7050/tcp orderer.example.com
This shows that the nodes exited with an error code, they all look the same, look below for a docker logs of the peer node.
So my final question is: How do I get the "First Network" Hyperledger sample peers to successfully join the channel?
Thanks in advance!
Update 1
I chose a bad code dump! Please use these links for logs/outputs.
Full ./byfn up output
Docker log output for peer0
Update 2
So I have been trying various things, it seems to not be a go related error but simply a "connection" error where go crashes upon trying to connect a peer to the channel. So the main question at hand is: Why are my docker instances not properly connecting to the channel?
Update 3
I have used Amazon Web Services to launch a Linux instance and re-created all my installation steps on this "fresh" instance. Everything worked on the first go (pun intended). Therefore I must conclude that it had to do with either my network settings or personal setup as these are the only parameters that changed.
As this works for me for now I will work with that. I am still open to suggestions and will keep an eye on this post!
Package versions
Hyperledger Fabric 1.4.0
Docker version 18.09.2, build 6247962
docker-compose version 1.13.0, build 1719ceb
go version go1.11 linux/amd64
npm: '6.4.1',
node -v: v8.15.0
I will suggest you to check two things: the memory available and the permissions in the "first-network" directory.
I attempt to deploy Pachyderm (a docker bigdata platform) on kubernetes. Limited by Pachyderm, I have to install kubernetes v1.2.2, an old version. I follow the guide here http://kubernetes.io/docs/getting-started-guides/docker/ to deploy Kubernetes on local server via docker. The guide can work with the kubernetes >=1.3.0, but when I use it to deploy kubernetes 1.2.2, I met some problems.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec38ae951f09 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube apiserver" 8 seconds ago Exited (255) 7 seconds ago k8s_apiserver.78ec1de_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_d26fc24e
55c1b13bb610 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" 8 seconds ago Up 8 seconds k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_1cb4c220
b9f0e5b3a7a9 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube scheduler" 9 seconds ago Up 8 seconds k8s_scheduler.fc12fcbe_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_e5065506
9cd613d272bc gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube apiserver" 9 seconds ago Exited (255) 8 seconds ago k8s_apiserver.78ec1de_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_c04426af
49fe2c409386 gcr.io/google_containers/etcd:2.2.1 "/usr/local/bin/etcd " 10 seconds ago Up 9 seconds k8s_etcd.7e452b0b_k8s-etcd-127.0.0.1_default_1df6a8b4d6e129d5ed8840e370203c11_a6f11fdb
5b208be18c71 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube controlle" 10 seconds ago Up 9 seconds k8s_controller-manager.70414b65_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_c377c5e9
df194f3cf663 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube proxy --m" 10 seconds ago Up 9 seconds k8s_kube-proxy.9a9f4853_k8s-proxy-127.0.0.1_default_5e5303a9d49035e9fad52bfc4c88edc8_63ec0b04
58b53ec28fbe gcr.io/google_containers/pause:2.0 "/pause" 10 seconds ago Up 9 seconds k8s_POD.6059dfa2_k8s-etcd-127.0.0.1_default_1df6a8b4d6e129d5ed8840e370203c11_21034b2e
df48fe4cdf0a gcr.io/google_containers/pause:2.0 "/pause" 10 seconds ago Up 9 seconds k8s_POD.6059dfa2_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_4867dbbc
fe6b74c2a881 gcr.io/google_containers/pause:2.0 "/pause" 10 seconds ago Up 9 seconds k8s_POD.6059dfa2_k8s-proxy-127.0.0.1_default_5e5303a9d49035e9fad52bfc4c88edc8_fad2c558
4c00ad498916 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube kubelet -" 25 seconds ago Up 24 seconds kubelet
From the docker container table, it can be observed that my apiserver is down when deploying kubernetes1.2.2. The restart interval of kubernetes apiserver obeys expontional backoff algorithm. But never work.
Then,
sv: batch/v1
mv: extensions/__internal
I0727 06:06:27.593708 1 genericapiserver.go:82] Adding storage destination for group batch
W0727 06:06:27.593745 1 server.go:383] No RSA key provided, service account token authentication disabled
F0727 06:06:27.593767 1 server.go:410] Invalid Authentication Config: open /srv/kubernetes/basic_auth.csv: no such file or directory
Please see docker logs of kubernetes apiserver here. Note that some authentication error occurred seems that the Kubernetes does not have required key to be permitted.Also see the controller manager log here. The controller manager wait for the apiserver, however the apiserver hasn't ran ever. The controller manager is also dump.
E0727 06:07:10.604801 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:11.604832 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:12.604752 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:13.604803 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:14.604332 1 nodecontroller.go:229] Error monitoring node status: Get http://127.0.0.1:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:14.604619 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:14.604861 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
F0727 06:07:14.604957 1 controllermanager.go:263] Failed to get api versions from server: timed out waiting for the condition
So for my question, how to solve this problem? The problem has troubled me for a long time.
====================================================================
Update:
With the help of Goblin and Lukie, I find the key problem is the Setup Pods is not triggered.
See the manifest of Kubernetes,
{
"name": "controller-manager",
"/hyperkube",
"controller-manager",
"--master=127.0.0.1:8080",
"--service-account-private-key-file=/srv/kubernetes/server.key",
"--root-ca-file=/srv/kubernetes/ca.crt",
"--min-resync-period=3m",
"--v=2"
],
"volumeMounts": [
{
"name": "data",
"mountPath": "/srv/kubernetes"
}
]
}
Option --service-account-private-key-file=/srv/kubernetes/server.key has been added in the manifest file, but it doesn't work. In other words, the controller-manager cannot find this file in the file system. This assumption is supported by following command.
docker exec a82d7f6e4d7d ls -l /srv/kubernetes
ls: cannot access /srv/kubernetes: No such file or directory
Next, we check whether the Setup Pod put the file in the docker volumn. Unfortunately, we find that the Setup Pod is not triggered and worked, therefore no cert file is written in the file system.
docker ps -a | grep setup
54afdd81349e gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" About a minute ago Up About a minute k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_a2edddca
6f714e034098 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" 4 minutes ago Exited (7) 2 minutes ago k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_0d7dab5b
8358f6644d94 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" 6 minutes ago Exited (7) 4 minutes ago k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_41e4c686
Is there any method to do further debug? Or is it a bug in Kubernetes version 1.2?
F0727 06:06:27.593767 1 server.go:410] Invalid Authentication Config: open /srv/kubernetes/basic_auth.csv: no such file or directory
You are missing the basic auth file /srv/kubernetes/basic_auth.csv either createa basic auth file or remove the configuration flag.
Kubernetes authentication
in fact it is W0727 06:06:27.593745 1 server.go:383] No RSA key provided, service account token authentication disabled that is more important in my opinion.
Seems like --service-account-private-key-file is missing on controller-manager so service tokens can not be properly generated.