kubeadm init failing on ARM64 - arm64
I'm trying to setup a single master cluster on some sopine64s (quad A53 with 2GB RAM) running Armbian 5.38 (Ubuntu 16.04 based). Kernel is 3.10.107-pine64.
Steps taken so far:
usual ip address, hostname, timezone, dns, etc config
apt upgrade
disable swap
set net.bridge.bridge-nf-call-iptables to 1 in sysctl.conf (intending to use weavenet)
install docker 1.13.1 (docker.io package)
install kubeadm, kubelet, kubectl v1.11
systemctl enable and start kubelet and docker
reboot
kubeadm config images pull (all download ok)
Here's the output of kubeadm init:
I0712 18:58:42.149510 31708 feature_gate.go:230] feature gates: &{map[]}
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
I0712 18:58:42.301648 31708 kernel_validator.go:81] Validating kernel version
I0712 18:58:42.302621 31708 kernel_validator.go:96] Validating kernel config
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [sopine0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.16]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [sopine0 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [sopine0 localhost] and IPs [192.168.0.16 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- No internet connection is available so the kubelet cannot pull or find the following control plane images:
- k8s.gcr.io/kube-apiserver-arm64:v1.11.0
- k8s.gcr.io/kube-controller-manager-arm64:v1.11.0
- k8s.gcr.io/kube-scheduler-arm64:v1.11.0
- k8s.gcr.io/etcd-arm64:3.2.18
- You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
are downloaded locally and cached.
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster
If I look at the containers, the one for kube-apiserver is exiting and being recreated every few minutes. Here's it's log file:
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0712 07:06:39.855921 1 server.go:703] external host was not specified, using 192.168.0.16
I0712 07:06:39.856998 1 server.go:145] Version: v1.11.0
I0712 07:07:05.966337 1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0712 07:07:05.966598 1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0712 07:07:05.975261 1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0712 07:07:05.975630 1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0712 07:07:06.459185 1 master.go:234] Using reconciler: lease
W0712 07:07:30.376324 1 genericapiserver.go:319] Skipping API batch/v2alpha1 because it has no resources.
W0712 07:07:33.264038 1 genericapiserver.go:319] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0712 07:07:33.325028 1 genericapiserver.go:319] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0712 07:07:33.508270 1 genericapiserver.go:319] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0712 07:07:38.454808 1 genericapiserver.go:319] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/07/12 07:07:38 log.go:33: [restful/swagger] listing is available at https://192.168.0.16:6443/swaggerapi
[restful] 2018/07/12 07:07:38 log.go:33: [restful/swagger] https://192.168.0.16:6443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/07/12 07:07:48 log.go:33: [restful/swagger] listing is available at https://192.168.0.16:6443/swaggerapi
[restful] 2018/07/12 07:07:48 log.go:33: [restful/swagger] https://192.168.0.16:6443/swaggerui/ is mapped to folder /swagger-ui/
I0712 07:07:48.845592 1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0712 07:07:48.845818 1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0712 07:08:11.577474 1 serve.go:96] Serving securely on [::]:6443
I0712 07:08:11.578033 1 available_controller.go:278] Starting AvailableConditionController
I0712 07:08:11.578198 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0712 07:08:11.578033 1 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0712 07:08:11.581700 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0712 07:08:11.581449 1 crd_finalizer.go:242] Starting CRDFinalizer
I0712 07:08:11.581617 1 autoregister_controller.go:136] Starting autoregister controller
I0712 07:08:11.582060 1 cache.go:32] Waiting for caches to sync for autoregister controller
I0712 07:08:11.583450 1 controller.go:84] Starting OpenAPI AggregationController
I0712 07:08:11.584707 1 customresource_discovery_controller.go:199] Starting DiscoveryController
I0712 07:08:11.585112 1 naming_controller.go:284] Starting NamingConditionController
I0712 07:08:11.585243 1 establishing_controller.go:73] Starting EstablishingController
I0712 07:08:11.585336 1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0712 07:08:11.585379 1 controller_utils.go:1025] Waiting for caches to sync for crd-autoregister controller
I0712 07:08:13.059515 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41525: EOF
<above message repeats 9 more times on different ports in the 415xx range>
I0712 07:08:15.961160 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41566: EOF
I0712 07:08:16.582527 1 controller_utils.go:1032] Caches are synced for crd-autoregister controller
I0712 07:08:16.700615 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41564: EOF
<above message repeats 60 more times on different ports in the 41[5-7]xx range>
I0712 07:08:17.535106 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41720: EOF
I0712 07:08:17.560585 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0712 07:08:17.563061 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41723: EOF
I0712 07:08:17.577852 1 cache.go:39] Caches are synced for autoregister controller
I0712 07:08:17.596321 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41696: EOF
<above message repeats 6 more times on different ports in the 41[5-7]xx range>
I0712 07:08:17.686658 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41706: EOF
I0712 07:08:17.688440 1 trace.go:76] Trace[288588746]: "List /api/v1/services" (started: 2018-07-12 07:08:17.127883224 +0000 UTC m=+97.754900744) (total time: 560.373467ms):
Trace[288588746]: [560.004232ms] [559.9889ms] Listing from storage done
I0712 07:08:17.696643 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41726: EOF
<above message repeats 11 more times on different ports in the 41[5-7]xx range>
I0712 07:08:17.811279 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41716: EOF
I0712 07:08:17.831546 1 cache.go:39] Caches are synced for AvailableConditionController controller
I0712 07:08:17.850811 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41717: EOF
<above message repeats 11 more times on different ports in the 41[5-7]xx range>
I0712 07:08:18.303267 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41752: EOF
I0712 07:08:18.359750 1 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
I0712 07:08:18.386442 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41763: EOF
I0712 07:08:18.399648 1 logs.go:49] http: TLS handshake error from 192.168.0.16:41759: EOF
I0712 07:08:18.431038 1 trace.go:76] Trace[413119584]: "GuaranteedUpdate etcd3: *core.Pod" (started: 2018-07-12 07:08:17.845710035 +0000 UTC m=+98.472727763) (total time: 585.187661ms):
Trace[413119584]: [499.634456ms] [499.240097ms] Transaction prepared
I0712 07:08:18.432293 1 trace.go:76] Trace[838520449]: "Patch /api/v1/namespaces/kube-system/pods/kube-apiserver-sopine0/status" (started: 2018-07-12 07:08:17.845257845 +0000 UTC m=+98.472275323) (total time: 586.889091ms):
Trace[838520449]: [272.406761ms] [271.550004ms] About to check admission control
Trace[838520449]: [586.455609ms] [314.048848ms] Object stored in database
I0712 07:08:18.590379 1 controller.go:158] Shutting down kubernetes service endpoint reconciler
I0712 07:08:18.591681 1 available_controller.go:290] Shutting down AvailableConditionController
I0712 07:08:18.592066 1 autoregister_controller.go:160] Shutting down autoregister controller
I0712 07:08:18.592253 1 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0712 07:08:18.593252 1 crd_finalizer.go:254] Shutting down CRDFinalizer
I0712 07:08:18.593636 1 crdregistration_controller.go:143] Shutting down crd-autoregister controller
I0712 07:08:18.593831 1 establishing_controller.go:84] Shutting down EstablishingController
I0712 07:08:18.593962 1 naming_controller.go:295] Shutting down NamingConditionController
I0712 07:08:18.596110 1 customresource_discovery_controller.go:210] Shutting down DiscoveryController
I0712 07:08:18.596965 1 serve.go:136] Stopped listening on [::]:6443
I0712 07:08:18.597046 1 controller.go:90] Shutting down OpenAPI AggregationController
E0712 07:08:18.605877 1 memcache.go:147] couldn't get resource list for authorization.k8s.io/v1beta1: Get https://127.0.0.1:6443/apis/authorization.k8s.io/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.608345 1 memcache.go:147] couldn't get resource list for autoscaling/v1: Get https://127.0.0.1:6443/apis/autoscaling/v1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.610552 1 memcache.go:147] couldn't get resource list for autoscaling/v2beta1: Get https://127.0.0.1:6443/apis/autoscaling/v2beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.613608 1 memcache.go:147] couldn't get resource list for batch/v1: Get https://127.0.0.1:6443/apis/batch/v1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.616508 1 memcache.go:147] couldn't get resource list for batch/v1beta1: Get https://127.0.0.1:6443/apis/batch/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.619558 1 memcache.go:147] couldn't get resource list for certificates.k8s.io/v1beta1: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.620335 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:discovery: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:discovery: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.623207 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:basic-user: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:basic-user: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.630429 1 available_controller.go:311] v1beta1.extensions failed with: Put https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.extensions/status: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.632957 1 available_controller.go:311] v1beta1.batch failed with: Put https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.batch/status: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.634480 1 available_controller.go:311] v1.authorization.k8s.io failed with: Put https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1/apiservices/v1.authorization.k8s.io/status: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.636395 1 memcache.go:147] couldn't get resource list for networking.k8s.io/v1: Get https://127.0.0.1:6443/apis/networking.k8s.io/v1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.637222 1 available_controller.go:311] v1beta1.authentication.k8s.io failed with: Put https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.authentication.k8s.io/status: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.637426 1 available_controller.go:311] v1.authentication.k8s.io failed with: Put https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1/apiservices/v1.authentication.k8s.io/status: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.637987 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/admin: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/admin: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.638575 1 memcache.go:147] couldn't get resource list for policy/v1beta1: Get https://127.0.0.1:6443/apis/policy/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.648978 1 repair.go:73] unable to refresh the port block: Get https://127.0.0.1:6443/api/v1/services: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.649411 1 controller.go:192] unable to sync kubernetes service: Post https://127.0.0.1:6443/api/v1/namespaces: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.649511 1 controller.go:179] unable to create required kubernetes system namespace kube-system: Post https://127.0.0.1:6443/api/v1/namespaces: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.652296 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/edit: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/edit: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.653581 1 memcache.go:147] couldn't get resource list for rbac.authorization.k8s.io/v1: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.656058 1 repair.go:88] unable to refresh the service IP block: Get https://127.0.0.1:6443/api/v1/services: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.657312 1 controller.go:179] unable to create required kubernetes system namespace kube-public: Post https://127.0.0.1:6443/api/v1/namespaces: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.657317 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/view: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/view: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.659613 1 memcache.go:147] couldn't get resource list for rbac.authorization.k8s.io/v1beta1: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.663703 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:aggregate-to-admin: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.665261 1 memcache.go:147] couldn't get resource list for storage.k8s.io/v1: Get https://127.0.0.1:6443/apis/storage.k8s.io/v1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.666096 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:aggregate-to-edit: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:aggregate-to-edit: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.667801 1 memcache.go:147] couldn't get resource list for storage.k8s.io/v1beta1: Get https://127.0.0.1:6443/apis/storage.k8s.io/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.669445 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:aggregate-to-view: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:aggregate-to-view: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.670988 1 memcache.go:147] couldn't get resource list for admissionregistration.k8s.io/v1beta1: Get https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.672630 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:heapster: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:heapster: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.673420 1 memcache.go:147] couldn't get resource list for apiextensions.k8s.io/v1beta1: Get https://127.0.0.1:6443/apis/apiextensions.k8s.io/v1beta1?timeout=32s: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.674753 1 storage_rbac.go:193] unable to reconcile clusterrole.rbac.authorization.k8s.io/system:node: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:node: dial tcp 127.0.0.1:6443: connect: connection refused
E0712 07:08:18.675802 1 controller.go:160] no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
Apologies for what may seem like a dump of log files, but any help is appreciated.
Did some more digging and apparently this is a known issue with bare metal kubeadm deployments on all versions above 1.9.6. I've been able to successfully run init by downgrading the version.
Related
"transport: error while dialing: dial tcp: lookup orderer.ordererOrg.example.com: no such host
While creating a simple-network in Fabric, when creating the channels and joins of the peers and orders I get the following error: 2022-01-26 16:34:15.416 CET [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized Error: Error getting broadcast client: orderer client failed to connect to orderer.ordererOrg.example.com:7050: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp: lookup orderer.ordererOrg.example.com: no such host" create.sh(docker ps -a) join.sh through the docker inspection controls I noticed that containers are created without the ip address but only the port. docker-compose.yaml docker inspect test-network
Disable ipv6 for docker in Ubuntu 14.04
I have an issue with the docker daemon installed on an Ubuntu 14.04 VM. The logs reveal that ipv6 is enabled hence the docker seems to be listening on this ip address. Essentially, this effects Clair. I have made sure that ipv6 is disabled on the following recommendation here. I also disabled ipv6 in daemon.json as specified in Docker documentation. My docker version is Docker version 17.06.1-ce, build 874a737.
Docker daemon logs :
time="2018-02-20T20:33:17.736203462+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 20 01:4860:4860::8844]"
Clair logs:
2018/02/20 20:43:51 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp [::]:6060: connect: cannot assign requested address"; Reconnecting to {[::]:6060 <nil>}
2018/02/20 20:46:14 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp [::]:6060: connect: cannot assign requested address"; Reconnecting to {[::]:6060 <nil>}
It's trying to make an IPv6 connection, but the address is wrong. [::] is IN6ADDR_ANY, not an actual address you can connect to. Provide the correct address in your config.yaml. Did you mean to connect to localhost? api: # v3 grpc/RESTful API server address addr: "[::1]:6060"
Docker and weave on two hosts can't ping each other
OS: window7 virtualization tool: virtualbox virtual hypervisor: centos7 linux core as below [root#localhost ~]# uname -a Linux localhost.localdomain 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Two host IPs: 192.168.100.101 192.168.100.102 The steps: curl -L git.io/weave -o /usr/local/bin/weave chmod a+x /usr/local/bin/weave [root#localhost ~]# weave launch 192.168.100.102 WARNING: existing iptables rule '-A FORWARD -j REJECT --reject-with icmp-host-prohibited' will block name resolution via weaveDNS - please reconfigure your firewall. cannot locate running docker daemon Warning: unable to detect proxy TLS configuration. To enable TLS, launch the proxy with ‘weave launch’ and supply TLS options. To suppress this warning, supply the ‘–no-detect-tls’ option. 3227932d5be77917c4e0b780cafe1171287c1029637f2360ece580fe6239cb4f [root#localhost ~]# weave status Version: 2.1.1 (failed to check latest version - see logs; next check at 2017/11/28 19:18:07) Service: router Protocol: weave 1..2 Name: 06:e0:c4:68:0c:ae(localhost.localdomain) Encryption: disabled PeerDiscovery: enabled Targets: 1 Connections: 1 (1 failed) Peers: 1 TrustedSubnets: none Service: ipam Status: ready Range: 10.32.0.0/12 DefaultSubnet: 10.32.0.0/12 Service: dns Domain: weave.local. Upstream: 135.251.4.190, 135.251.38.218, 192.168.1.1 TTL: 1 Entries: 0 Service: proxy Address: unix:///var/run/weave/weave.sock Service: plugin (legacy) DriverName: weave [root#localhost ~]# weave version weave script 2.1.1 weave 2.1.1 As above , the Connections: 1 (1 failed) , the docker run on these two hosts can’t ping each other. And when I type weave version, it only showed weave script 2.1.1 and weave 2.1.1, not like other articles said it will show weaveexec and plugin or weavedns and so on. Is this the difference between versions or some mistakes I have made ? Please help , thank you very much! my docker log as below [root#localhost ~]# docker logs weave INFO: 2017/11/29 01:08:07.807752 Command line options: map[dns-effective-listen-address:172.17.0.1 nickname:localhost.localdomain dns-listen-address:172.17.0.1:53 ipalloc-range:10.32.0.0/12 status-addr:127.0.0.1:6782 weave-bridge:weave H:[unix:///var/run/weave/weave.sock] host-root:/host http-addr:127.0.0.1:6784 port:6783 proxy:true resolv-conf:/var/run/weave/etc/resolv.conf datapath:datapath docker-bridge:docker0 plugin:true] INFO: 2017/11/29 01:08:07.807841 weave 2.1.1 INFO: 2017/11/29 01:08:07.859209 Docker API on unix:///var/run/docker.sock: &[ApiVersion=1.24 GoVersion=go1.8.3 Os=linux BuildTime=2017-10-24T15:40:21.112972404+00:00 PkgVersion=docker-1.12.6-61.git85d7426.el7.centos.x86_64 Version=1.12.6 Arch=amd64 KernelVersion=3.10.0-693.5.2.el7.x86_64 GitCommit=85d7426/1.12.6] INFO: 2017/11/29 01:08:07.859520 Using docker bridge IP for DNS: 172.17.0.1 INFO: 2017/11/29 01:08:07.863781 proxy listening on unix:///var/run/weave/weave.sock INFO: 2017/11/29 01:08:08.940871 Bridge type is bridged_fastdp INFO: 2017/11/29 01:08:08.940885 Communication between peers is unencrypted. INFO: 2017/11/29 01:08:08.961891 Our name is 06:e0:c4:68:0c:ae(localhost.localdomain) INFO: 2017/11/29 01:08:08.962058 Restart/resume detected - using persisted peer list: [192.168.100.102] INFO: 2017/11/29 01:08:08.972210 Docker API on unix:///var/run/docker.sock: &[KernelVersion=3.10.0-693.5.2.el7.x86_64 PkgVersion=docker-1.12.6-61.git85d7426.el7.centos.x86_64 GoVersion=go1.8.3 Os=linux Arch=amd64 BuildTime=2017-10-24T15:40:21.112972404+00:00 Version=1.12.6 ApiVersion=1.24 GitCommit=85d7426/1.12.6] INFO: 2017/11/29 01:08:08.974990 Checking for pre-existing addresses on weave bridge INFO: 2017/11/29 01:08:09.009949 [allocator 06:e0:c4:68:0c:ae] Initialising with persisted data INFO: 2017/11/29 01:08:09.034491 Listening for DNS queries on 172.17.0.1 INFO: 2017/11/29 01:08:09.086102 Sniffing traffic on datapath (via ODP) INFO: 2017/11/29 01:08:09.114882 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:09.116392 Listening for HTTP control messages on 127.0.0.1:6784 INFO: 2017/11/29 01:08:09.116576 Listening for metrics requests on 127.0.0.1:6782 INFO: 2017/11/29 01:08:09.125917 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:08:09.151109 Waiting for Weave API Server... INFO: 2017/11/29 01:08:09.159548 Finished waiting for Weave API Server INFO: 2017/11/29 01:08:09.159706 Listening on /run/docker/plugins/weave.sock for global scope INFO: 2017/11/29 01:08:09.159811 Listening on /run/docker/plugins/weavemesh.sock for local scope INFO: 2017/11/29 01:08:09.159822 Creating default "weave" network INFO: 2017/11/29 01:08:09.462160 Discovered local MAC 06:e0:c4:68:0c:ae INFO: 2017/11/29 01:08:09.547179 Discovered local MAC 46:2b:0d:08:12:be INFO: 2017/11/29 01:08:09.554830 Discovered local MAC 0e:46:f3:dd:57:96 INFO: 2017/11/29 01:08:11.612424 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:11.614477 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:08:13.980824 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:13.982289 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:08:18.124543 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:18.125556 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:08:23.294574 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:23.322022 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:08:37.070537 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:37.073928 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:08:39.222651 Error checking version: Get https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=1.12.6&flag_kernel-version=3.10.0-693.5.2.el7.x86_64&os=linux&signature=fvXv9SDD9r8gjV6d2HrXkVdBv5U72%2BeXQ6NT2u0JkKc%3D&version=2.1.1: dial tcp: lookup checkpoint-api.weave.works on 135.252.166.21:53: read udp 192.168.100.101:34840->135.252.166.21:53: i/o timeout INFO: 2017/11/29 01:08:46.009136 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:08:46.011168 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:09:16.169210 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:09:16.171278 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:09:42.294136 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:09:42.296081 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:10:28.752091 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:10:28.756481 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:12:03.755330 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:12:03.760374 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:14:30.481453 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:14:30.486632 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:17:41.166716 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:17:41.168341 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:22:38.820826 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:22:38.829815 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host INFO: 2017/11/29 01:23:08.968136 Expired MAC 06:e0:c4:68:0c:ae at 06:e0:c4:68:0c:ae(localhost.localdomain) INFO: 2017/11/29 01:23:08.968199 Expired MAC 46:2b:0d:08:12:be at 06:e0:c4:68:0c:ae(localhost.localdomain) INFO: 2017/11/29 01:23:08.968219 Expired MAC 0e:46:f3:dd:57:96 at 06:e0:c4:68:0c:ae(localhost.localdomain) INFO: 2017/11/29 01:30:27.085406 ->[192.168.100.102:6783] attempting connection INFO: 2017/11/29 01:30:27.089200 ->[192.168.100.102:6783] error during connection attempt: dial tcp4 :0->192.168.100.102:6783: getsockopt: no route to host but on 192.168.100.101 , I can ping 192.168.100.102 [root#localhost ~]# ping 192.168.100.102 PING 192.168.100.102 (192.168.100.102) 56(84) bytes of data. 64 bytes from 192.168.100.102: icmp_seq=1 ttl=64 time=1.19 ms 64 bytes from 192.168.100.102: icmp_seq=2 ttl=64 time=1.05 ms 64 bytes from 192.168.100.102: icmp_seq=3 ttl=64 time=0.906 ms ^C --- 192.168.100.102 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.906/1.051/1.195/0.120 ms
The issue has been resolved and shared my experience here. As I described in my question [root#localhost ~]# weave launch 192.168.100.102 WARNING: existing iptables rule '-A FORWARD -j REJECT --reject-with icmp-host-prohibited' will block name resolution via weaveDNS - please reconfigure your firewall. cannot locate running docker daemon Warning: unable to detect proxy TLS configuration. To enable TLS, launch the proxy with ‘weave launch’ and supply TLS options. To suppress this warning, supply the ‘–no-detect-tls’ option. 3227932d5be77917c4e0b780cafe1171287c1029637f2360ece580fe6239cb4f Then on both hosts ,I run command [root#localhost ~]# iptables -F And then stop and rm all docker container and stop weave and then restart docker daemon and then on host 192.168.100.101 ,run command [root#localhost ~]# weave launch [root#localhost ~]# eval $(weave env) [root#localhost ~]# docker run --name bbox1 -itd busybox [root#localhost ~]# docker run --name bbox2 -itd busybox and on host 192.168.100.102 , run command [root#localhost ~]# weave launch 192.168.100.101 [root#localhost ~]# eval $(weave env) [root#localhost ~]# docker run --name bbox3 -itd busybox And then test [root#localhost ~]# docker exec bbox3 ping -c2 bbox1 PING bbox1 (10.32.0.1): 56 data bytes 64 bytes from 10.32.0.1: seq=0 ttl=64 time=0.940 ms 64 bytes from 10.32.0.1: seq=1 ttl=64 time=2.362 ms --- bbox1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.940/1.651/2.362 ms Done! so it seemed that the iptables rule is the root cause. I will learn more iptables rules later so that avoid merely using "iptables -F". Thanks for #Marc Carré very much.
it only showed weave script 2.1.1 and weave 2.1.1, not like other articles said it will show weaveexec and plugin or weavedns and so on This is expected with your version of Weave Net. In prior versions, Weave Net started different containers, but starting Weave Net 2.0, these have been merged together. See also: https://github.com/weaveworks/weave/blob/master/CHANGELOG.md#release-200 All of Weave Net now runs in one container Previously we had three separate containers for routing, Docker API proxy and Docker plugin. Running everything in one simplifies start-up and removes the need to detect various error conditions. #1642,#2897,#2936,#2945,#2946,#2951,#2960 https://www.weave.works/blog/weave-net-2-released In Weave Net 2.0, the various processes forming Weave Net were merged into a single process. and what follows. Would you mind pointing to the docs which caused the confusion, so that we could improve these?
Connection error when deploying chaincode
I just started using chaincode.
I am following step by step:
http://hyperledger-fabric.readthedocs.io/en/latest/Setup/Chaincode-setup/#running-the-chaincode
I am using Docker toolbox on Windows.
But when I start to run chaincode_example02, I get the following errors:
2016/09/15 14:05:53 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 0.0.0.0:7051: connectex: The requested address is not valid in its context."; Reconnecting to {"0.0.0.0:7051" <nil>}
2016/09/15 14:05:54 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 0.0.0.0:7051: connectex: The requested address is not valid in its context."; Reconnecting to {"0.0.0.0:7051" <nil>}
2016/09/15 14:05:55 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 0.0.0.0:7051: connectex: The requested address is not valid in its context."; Reconnecting to {"0.0.0.0:7051" <nil>}
Why?
Flanneld not working in Kubernetes Multinode Example
I am trying to follow this tutorial with two Vagrant instances: http://kubernetes.io/v1.0/docs/getting-started-guides/docker-multinode.html After setting up master and worker node I tried to connect to the service's IP of a simple nginx-service from the master. But it looks like the kube-proxy cannot find the docker-container of the worker-node. The virtual IP of the service and the container-ip respond well on the worker-node That made me think of a misfunction of flanneld. Does anybody know how I could track down this error? Any help is appreciated! Thanks in advance Best, Johannes Output of Kube-Proxy container I1016 20:53:42.829290 1 proxysocket.go:130] Accepted TCP connection from 10.0.2.15:51774 to 10.0.2.15:40197 E1016 20:53:43.829575 1 proxysocket.go:99] Dial failed: dial tcp 10.1.12.3:80: i/o timeout E1016 20:53:45.825473 1 proxysocket.go:99] Dial failed: dial tcp 10.1.12.3:80: no route to host E1016 20:53:48.825556 1 proxysocket.go:99] Dial failed: dial tcp 10.1.12.3:80: no route to host E1016 20:53:51.825627 1 proxysocket.go:99] Dial failed: dial tcp 10.1.12.3:80: no route to host E1016 20:53:51.825710 1 proxysocket.go:133] Failed to connect to balancer: failed to connect to an endpoint.
This looks like https://github.com/kubernetes/kubernetes/issues/14426. Try upgrading flannel to 0.5.3.