I am trying out knative sequence/parallel flows.
I started with sequence example mentioned in official website for release v1.1
I created steps, sequence and pingSource as mentioned in the document but pingSource failed because sequence was not up.
Sequence has below exception
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UpdateFailed 8s (x6 over 8s) sequence-controller Failed to update status for "sequence": Sequence.flows.knative.dev "sequence" is invalid: [status.channelStatuses: Invalid value: "null": status.channelStatuses in body must be of type array: "null", status.subscriptionStatuses: Invalid value: "null": status.subscriptionStatuses in body must be of type array: "null"]
Warning UpdateFailed 3s (x5 over 8s) sequence-controller Failed to update status for "sequence": Sequence.flows.knative.dev "sequence" is invalid: [status.subscriptionStatuses: Invalid value: "null": status.subscriptionStatuses in body must be of type array: "null", status.channelStatuses: Invalid value: "null": status.channelStatuses in body must be of type array: "null"]
but steps are running fine
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/first ExternalName <none> first.varadhi.example.com 80/TCP 9m46s
service/first-00001 ClusterIP 10.96.116.201 <none> 80/TCP 9m51s
service/first-00001-private ClusterIP 10.96.155.146 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 9m51s
service/second ExternalName <none> second.varadhi.example.com 80/TCP 9m45s
service/second-00001 ClusterIP 10.96.208.230 <none> 80/TCP 9m51s
service/second-00001-private ClusterIP 10.96.171.83 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 9m51s
service/third ExternalName <none> third.varadhi.example.com 80/TCP 9m45s
service/third-00001 ClusterIP 10.96.131.110 <none> 80/TCP 9m51s
service/third-00001-private ClusterIP 10.96.55.219 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 9m51s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/first-00001-deployment 0/0 0 0 9m52s
deployment.apps/second-00001-deployment 0/0 0 0 9m52s
deployment.apps/third-00001-deployment 0/0 0 0 9m52s
NAME DESIRED CURRENT READY AGE
replicaset.apps/first-00001-deployment-594dc84cb8 0 0 0 9m52s
replicaset.apps/second-00001-deployment-79d9f8b7b8 0 0 0 9m52s
replicaset.apps/third-00001-deployment-7479456fdf 0 0 0 9m51s
NAME URL AGE READY REASON
channel.messaging.knative.dev/varadhi-inmem-channel 26h Unknown NewObservedGenFailure
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/first http://first.varadhi.example.com first-00001 first-00001 Unknown IngressNotConfigured
service.serving.knative.dev/second http://second.varadhi.example.com second-00001 second-00001 Unknown IngressNotConfigured
service.serving.knative.dev/third http://third.varadhi.example.com third-00001 third-00001 Unknown IngressNotConfigured
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
revision.serving.knative.dev/first-00001 first 1 True 0 0
revision.serving.knative.dev/second-00001 second 1 True 0 0
revision.serving.knative.dev/third-00001 third 1 True 0 0
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/first first-00001 first-00001 True
configuration.serving.knative.dev/second second-00001 second-00001 True
configuration.serving.knative.dev/third third-00001 third-00001 True
NAME URL READY REASON
route.serving.knative.dev/first http://first.varadhi.example.com Unknown IngressNotConfigured
route.serving.knative.dev/second http://second.varadhi.example.com Unknown IngressNotConfigured
route.serving.knative.dev/third http://third.varadhi.example.com Unknown IngressNotConfigured
After spending ample time on knative sequence, I decided to try out knative parallel
I referred official documentation of parallel for v1.1 and used multiple-branches examples
I created filters, transformers, parallel and pingSource but here also parallel didn't come up with below exception
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UpdateFailed 6m37s (x18 over 17m) parallel-controller Failed to update status for "odd-even-parallel": Parallel.flows.knative.dev "odd-even-parallel" is invalid: status.branchStatuses: Invalid value: "null": status.branchStatuses in body must be of type array: "null"
Here also parallel and sequence are up and running fine.
Has anyone faced similar issues or am I missing anything from official documentation ?
Environment info:
Using Kind Cluster
Using local docker registry (have bypassed digest check on images)
namespace : varadhi
Serving and Eventing CRDS are of knative v1.1
Edit #1
I have not created any channels explicitly and I do not see any channels being created implicitly by controller either
kubectl get channel -n varadhi
No resources found in varadhi namespace.
Also my default channel is InMemoryChannel
anil.gowda#faas-dev-kafka-8420816:~/knative$ kubectl get configmaps -n knative-eventing default-ch-webhook -o yaml
apiVersion: v1
data:
default-ch-config: |
clusterDefault:
apiVersion: messaging.knative.dev/v1
kind: InMemoryChannel
namespaceDefaults:
varadhi:
apiVersion: messaging.knative.dev/v1
kind: InMemoryChannel
kind: ConfigMap
Parallel :
Example Used : https://github.com/knative/docs/tree/main/code-samples/eventing/parallel/multiple-branches
Status
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/even-filter ExternalName <none> even-filter.varadhi.example.com 80/TCP 4d19h
service/even-filter-00001 ClusterIP 10.96.85.252 <none> 80/TCP 4d19h
service/even-filter-00001-private ClusterIP 10.96.98.109 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 4d19h
service/even-transformer ExternalName <none> even-transformer.varadhi.example.com 80/TCP 4d19h
service/even-transformer-00001 ClusterIP 10.96.152.53 <none> 80/TCP 4d19h
service/even-transformer-00001-private ClusterIP 10.96.130.58 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 4d19h
service/event-display ExternalName <none> event-display.varadhi.example.com 80/TCP 4d19h
service/event-display-00001 ClusterIP 10.96.237.175 <none> 80/TCP 4d19h
service/event-display-00001-private ClusterIP 10.96.81.3 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 4d19h
service/odd-filter ExternalName <none> odd-filter.varadhi.example.com 80/TCP 4d19h
service/odd-filter-00001 ClusterIP 10.96.84.239 <none> 80/TCP 4d19h
service/odd-filter-00001-private ClusterIP 10.96.16.17 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 4d19h
service/odd-transformer ExternalName <none> odd-transformer.varadhi.example.com 80/TCP 4d19h
service/odd-transformer-00001 ClusterIP 10.96.61.11 <none> 80/TCP 4d19h
service/odd-transformer-00001-private ClusterIP 10.96.203.185 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 4d19h
service/odd-transformer-00002 ClusterIP 10.96.115.147 <none> 80/TCP 4d19h
service/odd-transformer-00002-private ClusterIP 10.96.235.117 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 4d19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/even-filter-00001-deployment 0/0 0 0 4d19h
deployment.apps/even-transformer-00001-deployment 0/0 0 0 4d19h
deployment.apps/event-display-00001-deployment 0/0 0 0 4d19h
deployment.apps/odd-filter-00001-deployment 0/0 0 0 4d19h
deployment.apps/odd-transformer-00001-deployment 0/0 0 0 4d19h
deployment.apps/odd-transformer-00002-deployment 0/0 0 0 4d19h
NAME DESIRED CURRENT READY AGE
replicaset.apps/even-filter-00001-deployment-6b7bdd866f 0 0 0 4d19h
replicaset.apps/even-transformer-00001-deployment-666bf9d776 0 0 0 4d19h
replicaset.apps/event-display-00001-deployment-758c9f7595 0 0 0 4d19h
replicaset.apps/odd-filter-00001-deployment-c86bd4799 0 0 0 4d19h
replicaset.apps/odd-transformer-00001-deployment-6bf46bc88f 0 0 0 4d19h
replicaset.apps/odd-transformer-00002-deployment-5c5f7b8b75 0 0 0 4d19h
NAME URL AGE READY REASON
parallel.flows.knative.dev/odd-even-parallel 4d19h
NAME URL AGE READY REASON
NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/even-filter even-filter-00001 even-filter-00001 True
configuration.serving.knative.dev/even-transformer even-transformer-00001 even-transformer-00001 True
configuration.serving.knative.dev/event-display event-display-00001 event-display-00001 True
configuration.serving.knative.dev/odd-filter odd-filter-00001 odd-filter-00001 True
configuration.serving.knative.dev/odd-transformer odd-transformer-00002 odd-transformer-00002 True
NAME URL READY REASON
route.serving.knative.dev/even-filter http://even-filter.varadhi.example.com Unknown IngressNotConfigured
route.serving.knative.dev/even-transformer http://even-transformer.varadhi.example.com Unknown IngressNotConfigured
route.serving.knative.dev/event-display http://event-display.varadhi.example.com Unknown IngressNotConfigured
route.serving.knative.dev/odd-filter http://odd-filter.varadhi.example.com Unknown IngressNotConfigured
route.serving.knative.dev/odd-transformer http://odd-transformer.varadhi.example.com Unknown IngressNotConfigured
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/even-filter http://even-filter.varadhi.example.com even-filter-00001 even-filter-00001 Unknown IngressNotConfigured
service.serving.knative.dev/even-transformer http://even-transformer.varadhi.example.com even-transformer-00001 even-transformer-00001 Unknown IngressNotConfigured
service.serving.knative.dev/event-display http://event-display.varadhi.example.com event-display-00001 event-display-00001 Unknown IngressNotConfigured
service.serving.knative.dev/odd-filter http://odd-filter.varadhi.example.com odd-filter-00001 odd-filter-00001 Unknown IngressNotConfigured
service.serving.knative.dev/odd-transformer http://odd-transformer.varadhi.example.com odd-transformer-00002 odd-transformer-00002 Unknown IngressNotConfigured
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
revision.serving.knative.dev/even-filter-00001 even-filter 1 True 0 0
revision.serving.knative.dev/even-transformer-00001 even-transformer 1 True 0 0
revision.serving.knative.dev/event-display-00001 event-display 1 True 0 0
revision.serving.knative.dev/odd-filter-00001 odd-filter 1 True 0 0
revision.serving.knative.dev/odd-transformer-00001 odd-transformer 1 False ImagePullBackOff 0
revision.serving.knative.dev/odd-transformer-00002 odd-transformer 2 True 0 0
NAME SINK SCHEDULE AGE READY REASON
pingsource.sources.knative.dev/ping-source */1 * * * * 4d19h False NotFound
Edit #2
I installed few more eventing crds (eventing.yaml)
Few changes that I could see now.
Channels are getting created and their status is true
NAME URL AGE READY REASON
inmemorychannel.messaging.knative.dev/odd-even-parallel-kn-parallel http://odd-even-parallel-kn-parallel-kn-channel.varadhi.svc.cluster.local 73m True
inmemorychannel.messaging.knative.dev/odd-even-parallel-kn-parallel-0 http://odd-even-parallel-kn-parallel-0-kn-channel.varadhi.svc.cluster.local 73m True
inmemorychannel.messaging.knative.dev/odd-even-parallel-kn-parallel-1 http://odd-even-parallel-kn-parallel-1-kn-channel.varadhi.svc.cluster.local 73m True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-0 http://sequence-kn-sequence-0-kn-channel.varadhi.svc.cluster.local 70m True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-1 http://sequence-kn-sequence-1-kn-channel.varadhi.svc.cluster.local 50m True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-2 http://sequence-kn-sequence-2-kn-channel.varadhi.svc.cluster.local 50m True
but sequence and parallel are not in true state yet, they say subscriptionsNotReady
$ kubectl -n varadhi get sequence
NAME URL AGE READY REASON
sequence http://sequence-kn-sequence-0-kn-channel.varadhi.svc.cluster.local 71m Unknown SubscriptionsNotReady
$ kubectl -n varadhi get parallel
NAME URL AGE READY REASON
odd-even-parallel http://odd-even-parallel-kn-parallel-kn-channel.varadhi.svc.cluster.local 5d21h False SubscriptionsNotReady
On debugging further I could see below message in sequence
Ready:
Last Transition Time: 2022-01-12T08:19:17Z
Message: Failed to get subscription status: subscription "sequence-kn-sequence-0" not present in channel "sequence-kn-sequence-0" subscriber's list
Reason: SubscriptionNotMarkedReadyByChannel
Status: Unknown
Type: Ready
Subscription:
Name: sequence-kn-sequence-0
Namespace: varadhi
and on looking into channel, it gives below message
Last Transition Time: 2022-01-12T08:19:17Z
Message: The status of Dispatcher Deployment is False: MinimumReplicasUnavailable : Deployment does not have minimum availability.
Reason: DispatcherDeploymentFalse
Severity: Info
Status: False
Type: DispatcherReady
Looks like channel is not able to fetch service deployment status
I also installed Kourier but I cannot see external ip. Note that I am running my own kind cluster for this.
$ kubectl --namespace kourier-system get service kourier
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kourier LoadBalancer 10.96.112.238 <pending> 80:32002/TCP,443:32733/TCP 73m
$
$ kubectl get pods -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-d4cd7dfd5-mcxsl 1/1 Running 0 7d18h
autoscaler-69689d8b7-rx75h 1/1 Running 0 7d18h
controller-766f74d9f8-fwdk9 1/1 Running 0 7d18h
domain-mapping-7dbbb5c7d-xk5m5 1/1 Running 0 7d18h
domainmapping-webhook-747f79dbdc-qm5nn 1/1 Running 0 7d18h
net-kourier-controller-5657664b99-zr9cj 1/1 Running 0 73m
webhook-8f6866966-8z8tt 1/1 Running 0 7d18h
Few changes in services post installing kourier
$ kubectl get service.serving.knative.dev -n varadhi
NAME URL LATESTCREATED LATESTREADY READY REASON
even-filter http://even-filter.varadhi.example.com even-filter-00001 even-filter-00001 Unknown
even-transformer http://even-transformer.varadhi.example.com even-transformer-00001 even-transformer-00001 Unknown
event-display http://event-display.varadhi.example.com event-display-00001 event-display-00001 Unknown
first http://first.varadhi.example.com first-00001 first-00001 Unknown
odd-filter http://odd-filter.varadhi.example.com odd-filter-00001 odd-filter-00001 Unknown
odd-transformer http://odd-transformer.varadhi.example.com odd-transformer-00002 odd-transformer-00002 Unknown
second http://second.varadhi.example.com second-00001 second-00001 Unknown
third http://third.varadhi.example.com third-00001 third-00001 Unknown
IngressNotConfigured status goes away.
for the Sequence example I think there might be something wrong with your default channel configuration.
Sequences create channels to communicate between every step, it is expected that three channels are created for that example:
$ kubectl get channel
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-0 http://sequence-kn-sequence-0-kn-channel.default.svc.cluster.local 56s True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-1 http://sequence-kn-sequence-1-kn-channel.default.svc.cluster.local 56s True
inmemorychannel.messaging.knative.dev/sequence-kn-sequence-2 http://sequence-kn-sequence-2-kn-channel.default.svc.cluster.local 56s True
Can you check what channels are created for you and their status?
Can you also make sure your default channel is properly setup? See:
https://knative.dev/docs/eventing/channels/channel-types-defaults/
Can you also post here the Parallel you are using?
Status is up to the controller to fill, that one sounds like an issue.
At both of your outputs for sequence and parallel an issue with services can be found:
service.serving.knative.dev/even-filter http://even-filter.varadhi.example.com even-filter-00001 even-filter-00001 Unknown IngressNotConfigured
Although Knative Eventing does not depend on Serving, the examples for sequence/parallel requires it to be properly installed because they use serverless services:
sequence example: event-display, first, second and third
parallel example: event-display, even-filter, even-transformer, odd-filter even-transformer.
Eventing can use regular kubernetes services instead of knative services, but I think the best way to make the examples work for you is making sure Knative Serving works as expected.
Did you configure a network provider for Knative Serving?
If you did not, can you go through this step:
https://knative.dev/docs/install/serving/install-serving-with-yaml/#install-a-networking-layer
If you are in doubt about which one to choose I would go for Kourier, which is maintained by the Knative project.
Related
I'm starting with a new project, default VPC, and forked fluxcd/flux2-kustomize-helm-example github repository.
When I attempted to flux bootstrap into a clean new PRIVATE Autopilot K8s cluster, nothing became available (see below). The pods were stuck at ImagePullBackOff and the log traces looked like everything was in airplane mode.
I suspect I need to open up a Cloud NAT access to ghcr.io/fluxcd/helm-controller, github.com/fluxcd, et. al. unless there is a fluxcd mirror within gcr.io.
NAME READY STATUS RESTARTS AGE
pod/helm-controller-57ff7dd7b5-nnpm8 0/1 ImagePullBackOff 0 4m50s
pod/kustomize-controller-9f9bf46d9-wzcdr 0/1 ImagePullBackOff 0 4m50s
pod/notification-controller-64496c6d67-g6wpx 0/1 ImagePullBackOff 0 4m50s
pod/source-controller-7467658dcb-t6bsp 0/1 ImagePullBackOff 0 4m50s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notification-controller ClusterIP 10.42.1.103 <none> 80/TCP 4m51s
service/source-controller ClusterIP 10.42.3.58 <none> 80/TCP 4m51s
service/webhook-receiver ClusterIP 10.42.1.217 <none> 80/TCP 4m51s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/helm-controller 0/1 1 0 4m51s
deployment.apps/kustomize-controller 0/1 1 0 4m51s
deployment.apps/notification-controller 0/1 1 0 4m51s
deployment.apps/source-controller 0/1 1 0 4m51s
NAME DESIRED CURRENT READY AGE
replicaset.apps/helm-controller-57ff7dd7b5 1 1 0 4m50s
replicaset.apps/kustomize-controller-9f9bf46d9 1 1 0 4m50s
replicaset.apps/notification-controller-64496c6d67 1 1 0 4m50s
replicaset.apps/source-controller-7467658dcb 1 1 0 4m50s
This question belongs to the problem
Deployment of Ingress-controler with Helm failed
but i want also understand more about the background.
Basic situation is: Pod creation fails with error:
{"err":"Get "https://10.96.0.1:443/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx-admission": dial tcp 10.96.0.1:443: i/o timeout","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:232","time":"2022-02-22T10:47:49Z"}
i can see that the pod tries to get something from my kubernetes cluster-IP which listen on 443:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 121d
default nextcloud-service ClusterIP 10.98.154.93 <none> 82/TCP 13d
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 120d
My questions are:
Can i - somehow - check with a command if the URL-path really exist?
When will this secret be created, and how can i observe this?
Can i manipulate the cluster to use another port for this, like 8080 (non secure) or so?
When i check my secrets with command kubectl get secrets -A i see following results
NAMESPACE NAME TYPE DATA AGE
default default-token-95b8q kubernetes.io/service-account-token 3 122d
ingress-nginx default-token-fbvmd kubernetes.io/service-account-token 3 21h
ingress-nginx ingress-nginx-admission-token-cdfbf kubernetes.io/service-account-token 3 11m
can i somehow tell the deployment script (in values.yaml) the exact name of this secret?
Hi I have two virtual machine in a local server with ubuntu 20.04 and i want to build a small cluster for my microservices. I ran the following step to setup my cluster but I got issue with calico-nodes. They are running with 0/1/
master.domain.com
ubuntu 20.04
docker --version = Docker version 20.10.7, build f0df350
kubectl version = Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
worker.domain.com
ubuntu 20.04
docker --version = Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2
kubectl version = Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
STEP-1
In the master.domain.com virtual machine I run the following commands
sudo kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml
kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-7f4f5bf95d-gnll8 1/1 Running 0 38s 192.168.29.195 master <none> <none>
kube-system calico-node-7zmtm 1/1 Running 0 38s 195.251.3.255 master <none> <none>
kube-system coredns-74ff55c5b-ltn9g 1/1 Running 0 3m49s 192.168.29.193 master <none> <none>
kube-system coredns-74ff55c5b-nkhzf 1/1 Running 0 3m49s 192.168.29.194 master <none> <none>
kube-system etcd-kubem 1/1 Running 0 4m6s 195.251.3.255 master <none> <none>
kube-system kube-apiserver-kubem 1/1 Running 0 4m6s 195.251.3.255 master <none> <none>
kube-system kube-controller-manager-kubem 1/1 Running 0 4m6s 195.251.3.255 master <none> <none>
kube-system kube-proxy-2cr2x 1/1 Running 0 3m49s 195.251.3.255 master <none> <none>
kube-system kube-scheduler-kubem 1/1 Running 0 4m6s 195.251.3.255 master <none> <none>
STEP-2
In the worker.domain.com virtual machine I run the following commands
sudo kubeadm join 195.251.3.255:6443 --token azuist.xxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxx
STEP-3
In the master.domain.com virtual machine I run the following commands
kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-7f4f5bf95d-gnll8 1/1 Running 0 6m37s 192.168.29.195 master <none> <none>
kube-system calico-node-7zmtm 0/1 Running 0 6m37s 195.251.3.255 master <none> <none>
kube-system calico-node-wccnb 0/1 Running 0 2m19s 195.251.3.230 worker <none> <none>
kube-system coredns-74ff55c5b-ltn9g 1/1 Running 0 9m48s 192.168.29.193 master <none> <none>
kube-system coredns-74ff55c5b-nkhzf 1/1 Running 0 9m48s 192.168.29.194 master <none> <none>
kube-system etcd-kubem 1/1 Running 0 10m 195.251.3.255 master <none> <none>
kube-system kube-apiserver-kubem 1/1 Running 0 10m 195.251.3.255 master <none> <none>
kube-system kube-controller-manager-kubem 1/1 Running 0 10m 195.251.3.255 master <none> <none>
kube-system kube-proxy-2cr2x 1/1 Running 0 9m48s 195.251.3.255 master <none> <none>
kube-system kube-proxy-kxw4m 1/1 Running 0 2m19s 195.251.3.230 worker <none> <none>
kube-system kube-scheduler-kubem 1/1 Running 0 10m 195.251.3.255 master <none> <none>
kubectl logs -n kube-system calico-node-7zmtm
...
...
2021-06-20 17:10:25.064 [INFO][56] monitor-addresses/startup.go 774: Using autodetected IPv4 address on interface eth0: 195.251.3.255/24
2021-06-20 17:10:34.862 [INFO][53] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.5s: avg=4ms longest=13ms ()
kubectl logs -n kube-system calico-node-wccnb
...
...
2021-06-20 17:10:59.818 [INFO][55] felix/summary.go 100: Summarising 8 dataplane reconciliation loops over 1m3.6s: avg=3ms longest=13ms (resync-filter-v4,resync-nat-v4,resync-raw-v4)
2021-06-20 17:11:05.994 [INFO][51] monitor-addresses/startup.go 774: Using autodetected IPv4 address on interface br-9a88318dda68: 172.21.0.1/16
As you can see for both calico nodes I get 0/1 running, Why??
Any idea how to solve this problem?
Thank you
Got totally the same issue.
CentOS 8
kubectl kubeadm kubelet v1.22.3
docker-ce version 20.10.9
The only difference worth mention is that I have to comment line
- --port=0
in /etc/kubernetes/manifests/kube-scheduler.yaml or otherwise scheduler declared as unhealthy in
kubectl get componentstatuses
Kubernetes API is advertised on a public IP address.
Public IP address of control panel node is substituted with 42.42.42.42 in kubectl print-out;
Public IP address of worker node is substituted with 21.21.21.21
Public domain name (which is also a hostname on Control Panel node) is substituted with public-domain.work
>kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-5d995d45d6-rk9cq 1/1 Running 0 76m 192.168.231.193 public-domain.work <none> <none>
calico-node-qstxm 0/1 Running 0 76m 42.42.42.42 public-domain.work <none> <none>
calico-node-zmz5s 0/1 Running 0 75m 21.21.21.21 node1.public-domain.work <none> <none>
coredns-78fcd69978-5xsb2 1/1 Running 0 81m 192.168.231.194 public-domain.work <none> <none>
coredns-78fcd69978-q29fn 1/1 Running 0 81m 192.168.231.195 public-domain.work <none> <none>
etcd-public-domain.work 1/1 Running 3 82m 42.42.42.42 public-domain.work <none> <none>
kube-apiserver-public-domain.work 1/1 Running 3 82m 42.42.42.42 public-domain.work <none> <none>
kube-controller-manager-public-domain.work 1/1 Running 2 82m 42.42.42.42 public-domain.work <none> <none>
kube-proxy-5kkks 1/1 Running 0 81m 42.42.42.42 public-domain.work <none> <none>
kube-proxy-xsc66 1/1 Running 0 75m 21.21.21.21 node1.public-domain.work <none> <none>
kube-scheduler-public-domain.work 1/1 Running 1 (78m ago) 78m 42.42.42.42 public-domain.work <none> <none>
>kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
public-domain.work Ready control-plane,master 4h56m v1.22.3 42.42.42.42 <none> CentOS Stream 8 4.18.0-348.el8.x86_64 docker://20.10.9
node1.public-domain.work Ready <none> 4h50m v1.22.3 21.21.21.21 <none> CentOS Stream 8 4.18.0-348.el8.x86_64 docker://20.10.10
>kubectl logs -n kube-system calico-node-qstxm
2021-11-09 15:27:38.996 [INFO][86] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"eth1", Addrs:set.mapSet{}}
2021-11-09 15:27:38.996 [INFO][86] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"eth1", Addrs:set.mapSet{}}
2021-11-09 15:27:38.997 [INFO][86] felix/ipsets.go 130: Queueing IP set for creation family="inet" setID="this-host" setType="hash:ip"
2021-11-09 15:27:38.998 [INFO][86] felix/ipsets.go 785: Doing full IP set rewrite family="inet" numMembersInPendingReplace=7 setID="this-host"
2021-11-09 15:27:40.198 [INFO][86] felix/iface_monitor.go 201: Netlink address update. addr="here:is:some:ipv6:address:that:has:nothing:to:do:with:my:control:panel:server:public:ipv6" exists=true ifIndex=3 2021-11-09 15:27:40.198 [INFO][86] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{"fe80::9132:a0df:82d8:e26c":set.empty{}} ifaceName="eth1"
2021-11-09 15:27:40.198 [INFO][86] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"eth1", Addrs:set.mapSet{"here:is:some:ipv6:address:that:has:nothing:to:do:with:my:control:panel:server:public:ipv6":set.empty{}}}
2021-11-09 15:27:40.199 [INFO][86] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"eth1", Addrs:set.mapSet{"here:is:some:ipv6:address:that:has:nothing:to:do:with:my:control:panel:server:public:ipv6":set.empty{}}}
2021-11-09 15:27:40.199 [INFO][86] felix/ipsets.go 130: Queueing IP set for creation family="inet" setID="this-host" setType="hash:ip"
2021-11-09 15:27:40.200 [INFO][86] felix/ipsets.go 785: Doing full IP set rewrite family="inet" numMembersInPendingReplace=7 setID="this-host"
2021-11-09 15:27:48.010 [INFO][81] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 42.42.42.42/24
> kube-system calico-node-zmz5s
2021-11-09 15:25:56.669 [INFO][64] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="eth1"
2021-11-09 15:25:56.669 [INFO][64] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"eth1", Addrs:set.mapSet{}}
2021-11-09 15:25:56.669 [INFO][64] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"eth1", Addrs:set.mapSet{}}
2021-11-09 15:25:56.669 [INFO][64] felix/ipsets.go 130: Queueing IP set for creation family="inet" setID="this-host" setType="hash:ip"
2021-11-09 15:25:56.670 [INFO][64] felix/ipsets.go 785: Doing full IP set rewrite family="inet" numMembersInPendingReplace=7 setID="this-host"
2021-11-09 15:25:56.769 [INFO][64] felix/iface_monitor.go 201: Netlink address update. addr="here:is:some:ipv6:address:that:has:nothing:to:do:with:my:worknode:server:public:ipv6" exists=false ifIndex=3
2021-11-09 15:26:07.050 [INFO][64] felix/summary.go 100: Summarising 14 dataplane reconciliation loops over 1m1.7s: avg=5ms longest=11ms ()
2021-11-09 15:26:33.880 [INFO][59] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 21.21.21.21/24
Seemed that issue was in closed BGP port due to firewall.
This commands on master node solved it for me:
>firewall-cmd --add-port 179/tcp --zone=public --permanent
>firewall-cmd --reload
I’m trying to setup ingress using ambassador for my local cluster and working off this guide here https://kind.sigs.k8s.io/docs/user/ingress
but I receive an rpc and timeout error. Some things i’ve attempted in trying to fix include: Using the Ingress Nginx instead but I received same error, amending the dockerhub path by just having the username and image in the url and deleting and recreating the pods. I've also seen other previous questions and the solutions do not seem to work.
NAMESPACE NAME READY STATUS RESTARTS AGE
ambassador ambassador-operator-67668967b8-w28b2 0/1 ImagePullBackOff 0 28m
default bar-app 0/1 ErrImagePull 0 9m19s
default foo-app 0/1 ImagePullBackOff 0 9m19s
kube-system coredns-74ff55c5b-m7s8r 1/1 Running 0 38m
kube-system coredns-74ff55c5b-tgcdg 1/1 Running 0 38m
kube-system etcd-kind8-control-plane 1/1 Running 0 38m
kube-system kindnet-dch9w 1/1 Running 0 37m
kube-system kindnet-dm5gn 1/1 Running 0 38m
kube-system kindnet-sxxdk 1/1 Running 0 37m
kube-system kube-apiserver-kind8-control-plane 1/1 Running 0 38m
kube-system kube-controller-manager-kind8-control-plane 1/1 Running 0 38m
kube-system kube-proxy-n84kf 1/1 Running 0 38m
kube-system kube-proxy-twtsf 1/1 Running 0 37m
kube-system kube-proxy-zjq6t 1/1 Running 0 37m
kube-system kube-scheduler-kind8-control-plane 1/1 Running 0 38m
local-path-storage local-path-provisioner-78776bfc44-kkrht 1/1 Running 0 38m
This is the error log. kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'
kube-system 28m Normal Created pod/coredns-74ff55c5b-tgcdg Created container coredns
kube-system 28m Normal Created pod/coredns-74ff55c5b-m7s8r Created container coredns
kube-system 28m Normal Started pod/coredns-74ff55c5b-m7s8r Started container coredns
local-path-storage 28m Normal Started pod/local-path-provisioner-78776bfc44-kkrht Started container local-path-provisioner
local-path-storage 28m Normal Created pod/local-path-provisioner-78776bfc44-kkrht Created container local-path-provisioner
local-path-storage 28m Normal LeaderElection endpoints/rancher.io-local-path local-path-provisioner-78776bfc44-kkrht_c5300431-393d-4ce5-bee6-9fa03b2567e8 became leader
kube-system 28m Normal Started pod/coredns-74ff55c5b-tgcdg Started container coredns
ambassador 20m Normal ScalingReplicaSet deployment/ambassador-operator Scaled up replica set ambassador-operator-67668967b8 to 1
ambassador 20m Normal SuccessfulCreate replicaset/ambassador-operator-67668967b8 Created pod: ambassador-operator-67668967b8-w28b2
ambassador 20m Normal Scheduled pod/ambassador-operator-67668967b8-w28b2 Successfully assigned ambassador/ambassador-operator-67668967b8-w28b2 to kind8-worker
ambassador 15m Normal Pulling pod/ambassador-operator-67668967b8-w28b2 Pulling image "docker.io/datawire/ambassador-operator:v1.2.9"
ambassador 3s Warning Failed pod/ambassador-operator-67668967b8-w28b2 Error: ImagePullBackOff
ambassador 5m1s Normal BackOff pod/ambassador-operator-67668967b8-w28b2 Back-off pulling image "docker.io/datawire/ambassador-operator:v1.2.9"
ambassador 14m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Error: ErrImagePull
ambassador 19m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 18.214.230.110:443: i/o timeout
ambassador 17m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 3.211.199.249:443: i/o timeout
ambassador 16m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 54.236.165.68:443: i/o timeout
ambassador 14m Warning Failed pod/ambassador-operator-67668967b8-w28b2 Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 54.236.131.166:443: i/o timeout
default 38s Normal Scheduled pod/foo-app Successfully assigned default/foo-app to kind8-worker
default 38s Normal Scheduled pod/bar-app Successfully assigned default/bar-app to kind8-worker
default 37s Normal Pulling pod/bar-app Pulling image "hashicorp/http-echo:0.2.3"
This is the yaml.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ambassadorinstallations.getambassador.io
spec:
additionalPrinterColumns:
- JSONPath: .spec.version
name: VERSION
type: string
- JSONPath: .spec.updateWindow
name: UPDATE-WINDOW
type: integer
- JSONPath: .status.lastCheckTime
description: Last time checked
name: LAST-CHECK
type: string
- JSONPath: .status.conditions[?(#.type=='Deployed')].status
description: Indicates if deployment has completed
name: DEPLOYED
type: string
- JSONPath: .status.conditions[?(#.type=='Deployed')].reason
description: Reason for deployment completed
name: REASON
priority: 1
type: string
- JSONPath: .status.conditions[?(#.type=='Deployed')].message
description: Message for deployment completed
name: MESSAGE
priority: 1
type: string
- JSONPath: .status.deployedRelease.appVersion
description: Deployed version of Ambassador
name: DEPLOYED-VERSION
type: string
- JSONPath: .status.deployedRelease.flavor
description: Deployed flavor of Ambassador (OSS or AES)
name: DEPLOYED-FLAVOR
type: string
group: getambassador.io
names:
kind: AmbassadorInstallation
listKind: AmbassadorInstallationList
plural: ambassadorinstallations
singular: ambassadorinstallation
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: AmbassadorInstallation is the Schema for the ambassadorinstallations
API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: AmbassadorInstallationSpec defines the desired state of AmbassadorInstallation
properties:
baseImage:
description: An (optional) image to use instead of the image specified
in the Helm chart.
type: string
helmRepo:
description: An (optional) Helm repository.
type: string
installOSS:
description: 'Installs [Ambassador OSS](https://www.getambassador.io/docs/latest/topics/install/install-ambassador-oss/)
instead of [AES](https://www.getambassador.io/docs/latest/topics/install/).
Default is false which means it installs AES by default. TODO: 1.
AES/AOSS is not installed and the user installs using `installOSS:
true`, then we straightaway install AOSS. 2. AOSS is installed via
operator and the user sets `installOSS: false`, then we perform the
migration as detailed here - https://www.getambassador.io/docs/latest/topics/install/upgrade-to-edge-stack/
3. AES is installed and the user sets `installOSS: true`, then we
point users to the docs which gives them pointers on how to do
that themselves.'
type: boolean
logLevel:
description: 'An (optional) log level: debug, info...'
enum:
- info
- debug
- warn
- warning
- error
- critical
- fatal
type: string
updateWindow:
description: "`updateWindow` is an optional item that will control when
the updates can take place. This is used to force system updates to
happen late at night if that’s what the sysadmins want. \n * There
can be any number of `updateWindow` entries (separated by commas).
\ * `Never` turns off automatic updates even if there are other entries
in the comma-separated list. `Never` is used by sysadmins to disable
all updates during blackout periods by doing a `kubectl apply`
or using our Edge Policy Console to set this. * Each `updateWindow`
is in crontab format (see https://crontab.guru/) Some examples of
`updateWindows` are: - `* 0-6 * * * SUN`: every Sunday, from _0am_
to _6am_ - `* 5 1 * * *`: every first day of the month, at _5am_
* The Operator cannot guarantee minute time granularity, so specifying
\ a minute in the crontab expression can lead to some updates happening
\ sooner/later than expected."
type: string
version:
description: "We are using SemVer for the version number and it can
be specified with any level of precision and can optionally end in
`*`. These are interpreted as: \n * `1.0` = exactly version 1.0 *
`1.1` = exactly version 1.1 * `1.1.*` = version 1.1 and any bug fix
versions `1.1.1`, `1.1.2`, `1.1.3`, etc. * `2.*` = version 2.0 and
any incremental and bug fix versions `2.0`, `2.0.1`, `2.0.2`, `2.1`,
`2.2`, `2.2.1`, etc. * `*` = all versions. * `3.0-ea` = version `3.0-ea1`
and any subsequent EA releases on `3.0`. Also selects the final
3.0 once the final GA version is released. * `4.*-ea` = version `4.0-ea1`
and any subsequent EA release on `4.0`. Also selects the final GA
`4.0`. Also selects any incremental and bug fix versions `4.*` and
`4.*.*`. Also selects the most recent `4.*` EA release i.e., if
`4.0.5` is the last GA version and there is a `4.1-EA3`, then this
\ selects `4.1-EA3` over the `4.0.5` GA. \n You can find the reference
docs about the SemVer syntax accepted [here](https://github.com/Masterminds/semver#basic-comparisons)."
type: string
type: object
status:
description: AmbassadorInstallationStatus defines the observed state of
AmbassadorInstallation
properties:
conditions:
description: List of conditions the installation has experienced.
items:
description: AmbInsCondition defines an Ambassador installation condition,
as well as the last time there was a transition to this condition..
properties:
lastTransitionTime:
format: date-time
type: string
message:
type: string
reason:
type: string
status:
type: string
type:
type: string
required:
- status
- type
type: object
type: array
deployedRelease:
description: the currently deployed Helm chart
nullable: true
properties:
appVersion:
type: string
flavor:
type: string
manifest:
type: string
name:
type: string
version:
type: string
type: object
lastCheckTime:
description: Last time a successful update check was performed.
format: date-time
nullable: true
type: string
required:
- conditions
type: object
type: object
version: v2
versions:
- name: v2
served: true
storage: true
I figured this out. I'd previously disabled docker bridge0 using this entry under
/etc/docker/daemon.json
{
"iptables": false,
"bridge": "none"
}
To fixed i simply deleted and restarted docker:
systemctl restart docker
Environment information:
Computer detail: One master node and four slave nodes. All are CentOS Linux release 7.8.2003 (Core).
Kubernetes version: v1.18.0.
Zero to JupyterHub version: 0.9.0.
Helm version: v2.11.0
Recently, I try to deploy "Zero to Jupyterhub" on kubernetes. My jupyterhub config file such below:
config.yaml
proxy:
secretToken: "2fdeb3679d666277bdb1c93102a08f5b894774ba796e60af7957cb5677f40706"
service:
type: NodePort
nodePorts:
http: 30080
https: 30443
singleuser:
storage:
dynamic:
storageClass: local-storage
capacity: 10Gi
Note: I set the service type as NodePort, because I not have any cloud provider(deploy on my lab servers cluster), and I try using nginx-ingress also then got failure, that reason why I do not using LoadBalance.
But when I using this config file to install jupyterhub via Helm, I can not access jupyterhub from browser, even all Pods running. These pods detail like below:
kubectl get pod --namespace jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-8gxxk 1/1 Running 0 27m
continuous-image-puller-8tmdh 1/1 Running 0 27m
continuous-image-puller-lwdcx 1/1 Running 0 27m
continuous-image-puller-pszsr 1/1 Running 0 27m
hub-7b9cbbcf59-fbppq 1/1 Running 0 27m
proxy-6b699b54c8-2pxmb 1/1 Running 0 27m
user-scheduler-65f4cbb9b7-9vmfr 1/1 Running 0 27m
user-scheduler-65f4cbb9b7-lqfrh 1/1 Running 0 27m
and its services like this:
kubectl get service --namespace jhub
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hub ClusterIP 10.10.55.78 <none> 8081/TCP 28m
proxy-api ClusterIP 10.10.27.133 <none> 8001/TCP 28m
proxy-public NodePort 10.10.97.11 <none> 443:30443/TCP,80:30080/TCP 28m
Is seem to work well, right? (I guessed.) But the fact is that I can not use ip 10.10.97.11 to access the jupyter main page, and I did not get any external ip also.
So, my problems are:
Do my config have any wrong?
How to get an external ip?
Finally, thank you for save my day so much!
For NodePort service you will not get EXTERNAL-IP. You can not use the CLUSTER-IP to access it from outside the kubernetes cluster because CLUSTER-IP is for accessing it from inside the kubernetes cluster typically from another pod.For accessing from outside the kubernetes cluster you need to use NodeIP:NodePort where NodeIP is your kubernetes nodes IP address.