I'm trying to create a RabbitMQ instance using RabbitMQ cluster Kubernetes operator, but there is an issue with PersistentVolumeClaims. I'm running Kubernetes 1.18.8 using Docker Desktop for Windows.
I have installed the operator like this:
kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
I have created this very simple configuration for the instance according to the documentation:
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: nccrabbitmqcluster
It seems to create all of the objects it is supposed to create, but the pod gets stuck on pending state:
$ kubectl get all | grep rabbit
pod/nccrabbitmqcluster-server-0 0/1 Pending 0 14m
service/nccrabbitmqcluster ClusterIP 10.100.186.115 <none> 5672/TCP,15672/TCP 14m
service/nccrabbitmqcluster-nodes ClusterIP None <none> 4369/TCP,25672/TCP 14m
statefulset.apps/nccrabbitmqcluster-server 0/1 14m
There seems to be an unbound PVC according to the pod's events:
$ kubectl describe pod/nccrabbitmqcluster-server-0 | tail -n 5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "nccrabbitmqcluster-server-0": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "nccrabbitmqcluster-server-0": pod has unbound immediate PersistentVolumeClaims
According to the events of the PVC, it is waiting for a volume to be created:
$ kubectl describe pvc persistence-nccrabbitmqcluster-server-0
Name: persistence-nccrabbitmqcluster-server-0
Namespace: default
StorageClass: hostpath
Status: Pending
Volume:
Labels: app.kubernetes.io/component=rabbitmq
app.kubernetes.io/name=nccrabbitmqcluster
app.kubernetes.io/part-of=rabbitmq
Annotations: volume.beta.kubernetes.io/storage-provisioner: docker.io/hostpath
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: nccrabbitmqcluster-server-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 27s (x23 over 19m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "docker.io/hostpath" or manually created by system administrator
My understanding is that docker.io/hostpath is the correct provisioner:
$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
hostpath (default) docker.io/hostpath Delete Immediate false 20d
I can't see any PVs related to PCS:
$ kubectl get pv | grep rabbit
Why isn't the volume created automatically and what should I do?
Yes, your local hostpath can not work as dynamic volume provisioner. This operator needs an storageclassname which can dynamically create PVs.
In your case, your operator waiting continuously for PV to get created. In stead you can manually create an PV and PVC if you are doing in local machine.
Check this example - https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/multiple-disks/rabbitmq.yaml
If you are going to try any cloud provider like AWS then its pretty easy. Deploy EBS CSI driver in your cluster which will create an storageclass for you and that storageclass will provision dynamic volumes.
Related
My cluster includes: 1 master and 2 worker nodes. I created a pod using deployment yaml. The pod running successfully on the worker node 1, I can ping the pod's ip on worker nodes but I can't ping the ip address of the pod on the master. I tried to disable firewarlld, restart docker but not successfully. Please see my commands
[root#k8s-master ~]# kubectl get pods -o wide | grep qldv
qldv-liberty-8499dfcf67-55njr 1/1 Running 0 6m42s 10.40.0.2 worker-node1 <none> <none>
[root#k8s-master ~]# ping 10.40.0.2
PING 10.40.0.2 (10.40.0.2) 56(84) bytes of data.
From 10.32.0.1 icmp_seq=1 Destination Host Unreachable
From 10.32.0.1 icmp_seq=2 Destination Host Unreachable
From 10.32.0.1 icmp_seq=3 Destination Host Unreachable
[root#k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 43d v1.15.0
worker-node1 Ready <none> 42d v1.15.0
worker-node2 Ready <none> 42d v1.15.0
[root#k8s-master ~]# kubectl describe pod qldv-liberty-8499dfcf67-55njr
Name: qldv-liberty-8499dfcf67-55njr
Namespace: default
Priority: 0
Node: worker-node1/192.168.142.130
Start Time: Sat, 17 Aug 2019 20:05:57 +0700
Labels: app=qldv-liberty
pod-template-hash=8499dfcf67
Annotations: <none>
Status: Running
IP: 10.40.0.2
Controlled By: ReplicaSet/qldv-liberty-8499dfcf67
Containers:
qldv-liberty:
Container ID: docker://03636fb62d4cca0e41f4ad9f5a94b50cf371089ab5a0813ed802d02f4ac4b07a
Image: qldv-liberty
Image ID: docker://sha256:bd0d7ce1c07da5b9d398131b17da7a6931a9b7ae0673d19a6ec0c409416afc69
Port: 9080/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 17 Aug 2019 20:06:23 +0700
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-vtphv (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-vtphv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-vtphv
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 119s default-scheduler Successfully assigned default/qldv-liberty-8499dfcf67-55njr to worker-node1
Normal Pulled 96s kubelet, worker-node1 Container image "qldv-liberty" already present on machine
Normal Created 95s kubelet, worker-node1 Created container qldv-liberty
Normal Started 91s kubelet, worker-node1 Started container qldv-liberty
I have another app, it also has a pod that running on the worker node 1, and I can ping the pod's ip from master. But I don't know why it is impossible with above case.
Please help me !
I doubt that the cluster still exists, therefore I'd better share some troubleshooting tips:
Check status of all control plane components and node status. Ensure kube-proxy and network addon (flannel/calico/waive/etc) Pods exist on each node and in Ready state.
kubectl get deployments,daemonsets,pods,svc -A -o wide
There are several requirements for Kubernetes cluster, and it worth to check if they are satisfied.
Some useful information could be found in the control-plane component logs using
kubectl logs kube-component-name-pod -n kube-system
or kubelet logs using
journalctl -u kubelet
It's better to use well known images like nginx or mendhak/http-https-echo. They could be configured to listen any desired port and provide detailed information about requests in logs or in HTTP reply. It helps to exclude application/image related issues.
Check connectivity to Pod IP and Service ClusterIP within the same node first.
If worker node OS doesn't have necessary tools for troubleshooting (e.g container optimized images or coreOS), Pod with Ubuntu or Busybox image can be used for that. Creating Deployment or DaemonSet could help to schedule it on all nodes. Note that firewall or network issues can block kubectl exec connections to that pods.
If everything works fine within the same node, but connection to the Pod can't be established from another node it worth to check network addon status and nodes firewall configuration. Native Linux firewall helpers can interfere with iptables set of rules created by kube-proxy and block the connection.
Clusters created in public clouds may require additional routing, peering, cloud firewall or security groups configuration to allow full IPIP connectivity between cluster nodes, especially if they are created in different VPCs.
The next thing that worth to check is coredns/kube-dns health. They suppose to resolve to correct IP address cluster Services' names like servicename.namespacename.svc.cluster.local if requested using their pod IP addresses or kube-dns Service (it usually has IP address 10.96.0.10 in default kubeadm cluster configuration).
Solution for each problem could be found in another answers on StackExchange sites. Official documentation is another great source of information and also contains good examples.
I have installed kubernetes in AWS ec2 instance. I'm not using any minikube or openshift. I'm trying to install kamel on top of kubernetes to run my integration code. When I tried to run kamel install command its throwing below error,
Error: cannot find automatically a registry where to push images
When I tried running as root user below error is thrown,
Error: cannot get current namespace: open /root/.kube/config: no such file or directory
I'd like to know what registry I have to pass while running kamel install command. I have docker hub account with a demo repository. Should I pass something like,
kamel install --registry hubusername/reponame
What I'm not getting is after I passed value, I'm getting below success message,
Camel K installed in namespace default
When I tried to run a sample groovy script its getting hanged after following message
kamel run hello.groovy --dev
Integration "hello" created
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default camel-k-operator-587b579567-m26rs 0/1 Pending 0 30m <none> <none> <none> <none>
Name: camel-k-operator-587b579567-m26rs
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: camel.apache.org/component=operator
name=camel-k-operator
pod-template-hash=587b579567
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/camel-k-operator-587b579567
Containers:
camel-k-operator:
Image: docker.io/apache/camel-k:0.3.3
Port: <none>
Host Port: <none>
Command:
camel-k
Environment:
WATCH_NAMESPACE: default (v1:metadata.namespace)
OPERATOR_NAME: camel-k
POD_NAME: camel-k-operator-587b579567-m26rs (v1:metadata.name)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from camel-k-operator-token-prjhp (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
camel-k-operator-token-prjhp:
Type: Secret (a volume populated by a Secret)
SecretName: camel-k-operator-token-prjhp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 38s (x23 over 31m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Can you please help me out here? Thank you for your time.
If you installed a single node Kubernetes chances are your only node is a master node. Which is why Kubernetes won't schedule your job.
Check this by running:
kubectl get node
If your only node shows 'master' in its ROLES column - than you need to untaint it to allow scheduling:
kubectl taint nodes --all node-role.kubernetes.io/master-
Try to rerun you kamel job afer that.
Base question: When I try to use kube-apiserver on my master node, I get command not found error. How I can install/configure kube-apiserver? Any link to example will help.
$ kube-apiserver --enable-admission-plugins DefaultStorageClass
-bash: kube-apiserver: command not found
Details: I am new to Kubernetes and Docker and was trying to create StatefulSet with volumeClaimTemplates. My problem is that the automatic PVs are not created and I get this message in the PVC log: "persistentvolume-controller waiting for a volume to be created". I am not sure if I need to define DefaultStorageClass and so needed kube-apiserver to define it.
Name: nfs
Namespace: default
StorageClass: example-nfs
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner=example.com/nfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 3m (x2401 over 10h) persistentvolume-controller waiting for a volume to be created, either by external provisioner "example.com/nfs" or manually created by system administrator
Here is get pvc result:
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs Pending example-nfs 10h
And get storageclass:
$ kubectl describe storageclass example-nfs
Name: example-nfs
IsDefaultClass: No
Annotations: <none>
Provisioner: example.com/nfs
Parameters: <none>
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
How can I troubleshoot this issue (e.g. logs for why the storage was not created)?
You are asking two different questions here, one about kube-apiserver configuration, one about troubleshooting your StorageClass.
Here's an answer for your first question:
kube-apiserver is running as a Docker container on your master node. Therefore, the binary is within the container, not on your host system. It is started by the master's kubelet from a file located at /etc/kubernetes/manifests. kubelet is watching this directory and will start any Pod defined here as "static pods".
To configure kube-apiserver command line arguments you need to modify /etc/kubernetes/manifests/kube-apiserver.yaml on your master.
I'll refer to the question regarding the location of the api-server.
Basic answer (specific to the question title):
The kube apiserver is located on the master node (known as the control plane).
It can be executed:
1 ) Via the host's init system (like systemd).
2 ) As a pod (I'll explain below).
In both cases it will be located on the control plane (left side below):
If its running under systemD you can run: systemctl status api-server to see the path to the configuration (drop-in) file.
If it is running as pod you can view it under the kube-system namespace with all other control panel components (plus kube-proxy and maybe network solution like weave below):
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-lpdlc 1/1 Running 1 2d22h
coredns-f9fd979d6-vcs7g 1/1 Running 1 2d22h
etcd-my-master 1/1 Running 1 2d22h
kube-apiserver-my-master 1/1 Running 1 2d22h #<----Here
kube-controller-manager-my-master 1/1 Running 1 2d22h
kube-proxy-kh2lc 1/1 Running 1 2d22h
kube-scheduler-my-master 1/1 Running 1 2d22h
weave-net-59r5b 2/2 Running 3 2d22h
You can run:
kubectl describe pod/kube-apiserver-my-master -n kube-system
In order to get more details regarding the pod.
A bit more advanced answer:
(regarding the location of /etc/kubernetes/manifests)
Lets say we have no idea where to find the relevant path for the kube-api-server config file.
But we need to remember two important things:
1 ) The kube-api-server is running on the master node.
2 ) The Kubelet isn't running as pod and when the control plane components (plus kube-proxy) are executed as static pods - it is done by the Kubelet on the master node.
So we can start our journey for reaching the manifests path by investigating the Kubelet logs.
If the Kubelet is running for a long time it will be a very large file and we'll need to dump it somewhere and go to the begging - or if Kubelet was started 5 minutes ago we can run:
sudo journalctl -u kubelet --since -5m >> kubelet_5_minutes.log
And a quick search for "api-server" will bring us to the 2 lines below where the path of the manifests in mentioned:
my-master kubelet[71..]: 00:03:21 kubelet.go:261] Adding pod path: /etc/kubernetes/manifests
my-master kubelet[71..]: 00:03:21 kubelet.go:273] Watching apiserver
And also we can see that the Kubelet is trying to create the kube-apiserver pod under my-master node and inside the kube-system namespace:
my-master kubelet[71..]: 00:03:29.05 kubelet.go:1576] ..
Creating a mirror pod for "kube-apiserver-my-master_kube-system
To make the storage class "example-nfs" default, you need to run the below command:
kubectl patch storageclass example-nfs -p '{"metadata":
{"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'
I am trying to configure my Kubernetes cluster to use a local NFS server for persistent volumes.
I set up the PersistentVolume as follows:
apiVersion: v1
kind: PersistentVolume
metadata:
name: hq-storage-u4
namespace: my-ns
spec:
capacity:
storage: 10Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
path: /data/u4
server: 10.30.136.79
readOnly: false
The PV looks OK in kubectl
$ kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
hq-storage-u4 10Ti RWX Retain Released my-ns/pv-50g 49m
I then try to create the PersistentVolumeClaim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-50gb
namespace: my-ns
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
Kubectl shows the pvc status is Pending
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
pvc-50gb Pending 16m
When I try to add the volume to a deployment, I get the error:
[SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "pvc-50gb", which is unexpected., SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "pvc-50gb", which is unexpected., SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "pvc-50gb", which is unexpected.]
How to I get the pvc to a working state?
It turned out that I needed to put the IP (I also put the path) in quotes. After fixing that, the pvc goes to status Bound, and the pod can mount correctly.
I can't comment on your post so I'll just attempt to answer this.
I've encountered 2 kinds of errors when PVCs don't work on my NFS cluster. Installing a PV usually succeed, so the status message provided doesn't say much.
The annotation and spec of the PV and the PVC are dissimilar. This doesn't look like the case.
The node of the pod that uses the NFS resource cannot mount the resource. Try mount -t nfs 10.30.136.79:/data/u4 /mnt on the node that is supposed to mount the NFS resource. This should succeed. If this fails, it could be
The lack of mount permissions. Rectify /etc/exports in your NFS server.
A firewall blocking the NFS ports. Fix the firewall.
One more thing, a non-privileged user in the pod might have trouble writing to the NFS resource. The uid/gid of the NFS user in the pod must match the perms of the NFS resource.
Bonne chance!
I have a replication controller running with the following spec:
apiVersion: v1
kind: ReplicationController
metadata:
name: owncloud-controller
spec:
replicas: 1
selector:
app: owncloud
template:
metadata:
labels:
app: owncloud
spec:
containers:
- name: owncloud
image: adimania/owncloud9-centos7
ports:
- containerPort: 80
volumeMounts:
- name: userdata
mountPath: /var/www/html/owncloud/data
resources:
requests:
cpu: 400m
volumes:
- name: userdata
hostPath:
path: /opt/data
Now I run a hpa using autoscale command.
$ kubectl autoscale rc owncloud-controller --max=5 --cpu-percent=10
I have also started heapster using kubernetes run command.
$ kubectl run heapster --image=gcr.io/google_containers/heapster:v1.0.2 --command -- /heapster --source=kubernetes:http://192.168.0.103:8080?inClusterConfig=false --sink=log
After all this, the autoscaling never kicks in. From logs, it seems that the actual CPU utilization is not getting reported.
$ kubectl describe hpa owncloud-controller
Name: owncloud-controller
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Thu, 26 May 2016 14:24:51 +0530
Reference: ReplicationController/owncloud-controller/scale
Target CPU utilization: 10%
Current CPU utilization: <unset>
Min replicas: 1
Max replicas: 5
ReplicationController pods: 1 current / 1 desired
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
44m 8s 92 {horizontal-pod-autoscaler } Warning FailedGetMetrics failed to get CPU consumption and request: metrics obtained for 0/1 of pods
44m 8s 92 {horizontal-pod-autoscaler } Warning FailedComputeReplicas failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/1 of pods
What am I missing here?
Most probably heapster is running in a wrong namespace ("default"). HPA expects heapster to be in "kube-system" namespace. Please, add --namespace=kube-system to kubectl run heapster command.
I installed hepaster under the name space "kube-system" and it worked. After running heapster, make sure it's running before you use HPA for your application.
How to run Heapster with Kubernetes cluster
I put all files here https://gitlab.com/abushoeb/kubernetes/tree/master/heapster. They are collected from the official Kubernetes Repository and made minor changes.
How to run Heapster
Go to the directory heapster where you have grafana.yaml, heapster.yaml and influxdb.yaml and run following command
$ kubectl create -f .
How to stop Heapster
Go to the same heapster directory and then run following command
$ kubectl delete -f .
How to check Heapster is running
You can access heapster metric model from the pod where heapster is running to make sure heapster is working. It can be accessed via web browser by accessing http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/. The same result can be seen by executing following command.
$ curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/
If you see the list of metrics then heapster is running correctly. You can also browse grafana dashboard to see it (find the ip of the pod where grafana is running and the access it http://grafana-pod-ip:grafana-service-port).
Full documentation of Heapster Metric Model are available here.
Also just run ($ kubectl cluster-info) and see if it shows results like this:
Kubernetes master is running at https://cluster-ip:6443
Heapster is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/heapster
kubernetes-dashboard is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
monitoring-grafana is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
monitoring-influxdb is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
Check influxdb
You can also check influxdb if it has data in it. Install Influxdb Client on your local machine to get connected to infuxdb database.
$ influx -host <cluster-ip> -port <influxdb-service-port>
Some Sample influxdb queries
show databases
use db-name
show measurements
select value from "cpu/node_capacity"
Reference and Help
https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md
https://github.com/kubernetes/heapster/blob/master/docs/debugging.md
https://blog.kublr.com/how-to-utilize-the-heapster-influxdb-grafana-stack-in-kubernetes-for-monitoring-pods-4a553f4d36c9
http://www.dasblinkenlichten.com/installing-cadvisor-and-heapster-on-bare-metal-kubernetes/
http://blog.arungupta.me/kubernetes-monitoring-heapster-influxdb-grafana/