When Using NFS volume, container not starting in Kubernetes - docker

Question

i am using nfs for volume in kubernetes pod using deployment.
Below are details of all files.
Filename :- nfs-server.yaml
kind: Service
apiVersion: v1
metadata:
name: nfs-service
spec:
selector:
role: nfs
ports:
# Open the ports required by the NFS server
# Port 2049 for TCP
- name: tcp-2049
port: 2049
protocol: TCP
# Port 111 for UDP
- name: udp-111
port: 111
protocol: UDP
---
kind: Pod
apiVersion: v1
metadata:
name: nfs-server-pod
labels:
role: nfs
spec:
containers:
- name: nfs-server-container
image: cpuguy83/nfs-server
securityContext:
privileged: true
args:
# Pass the paths to share to the Docker image
- /exports
Both service and pod are running. Below is the output.
Now i have to use this in my web-server. Below is the details of the deployment file for web.
Filename :- deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: apache-service
spec:
selector:
app: apache
ports:
- protocol: TCP
port: 80
targetPort: 80
type: NodePort
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: apache-deployment
spec:
selector:
matchLabels:
app: apache
replicas: 1 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: apache
spec:
volumes:
- name: nfs-volume
nfs:
server: 10.99.56.195
path: /exports
containers:
- name: apache
image: mobingi/ubuntu-apache2-php7:7.2
ports:
- containerPort: 80
volumeMounts:
- name: nfs-volume
mountPath: /var/www/html
when i am running this file without volume everything works fine. But when i am running this with nfs than pod give following error.
kubectl describe pod apache-deployment-577ffcf9bd-p8s75
Give following output:-
Name: apache-deployment-577ffcf9bd-p8s75
Namespace: default
Priority: 0
Node: worker-node2/10.160.0.4
Start Time: Tue, 09 Jul 2019 09:53:39 +0000
Labels: app=apache
pod-template-hash=577ffcf9bd
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/apache-deployment-577ffcf9bd
Containers:
apache:
Container ID:
Image: mobingi/ubuntu-apache2-php7:7.2
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p9qdb (ro)
/var/www/html from nfs-volume (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nfs-volume:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 10.244.1.50
Path: /exports
ReadOnly: false
default-token-p9qdb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p9qdb
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m21s default-scheduler Successfully assigned default/apache-deployment-577ffcf9bd-p8s75 to worker-node2
Warning FailedMount 4m16s kubelet, worker-node2 MountVolume.SetUp failed for volume "nfs-volume" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/29114119-5815-442a-bb97-03fa491206a4/volumes/kubernetes.io~nfs/nfs-volume --scope -- mount -t nfs 10.244.1.50:/exports /var/lib/kubelet/pods/29114119-5815-442a-bb97-03fa491206a4/volumes/kubernetes.io~nfs/nfs-volume
Output: Running scope as unit: run-r3a55a8a3287448a59f7e4dbefa0333af.scope
mount.nfs: Connection timed out
Warning FailedMount 2m10s kubelet, worker-node2 MountVolume.SetUp failed for volume "nfs-volume" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/29114119-5815-442a-bb97-03fa491206a4/volumes/kubernetes.io~nfs/nfs-volume --scope -- mount -t nfs 10.244.1.50:/exports /var/lib/kubelet/pods/29114119-5815-442a-bb97-03fa491206a4/volumes/kubernetes.io~nfs/nfs-volume
Output: Running scope as unit: run-r5fe7befa141d4f989e14b291afa43208.scope
mount.nfs: Connection timed out
Warning FailedMount 2m3s (x2 over 4m18s) kubelet, worker-node2 Unable to mount volumes for pod "apache-deployment-577ffcf9bd-p8s75_default(29114119-5815-442a-bb97-03fa491206a4)": timeout expired waiting for volumes to attach or mount for pod "default"/"apache-deployment-577ffcf9bd-p8s75". list of unmounted volumes=[nfs-volume]. list of unattached volumes=[nfs-volume default-token-p9qdb]
Warning FailedMount 4s kubelet, worker-node2 MountVolume.SetUp failed for volume "nfs-volume" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/29114119-5815-442a-bb97-03fa491206a4/volumes/kubernetes.io~nfs/nfs-volume --scope -- mount -t nfs 10.244.1.50:/exports /var/lib/kubelet/pods/29114119-5815-442a-bb97-03fa491206a4/volumes/kubernetes.io~nfs/nfs-volume
Output: Running scope as unit: run-rd30c104228ae43df933839b6da469107.scope
mount.nfs: Connection timed out
Can Anyone please help to solve this problem.

Answer 1

Make sure there is no firewall between the nodes
Make sure nfs-utils installed on cluster nodes
Here is a blog posts about the docker image that you are using for nfs server, you need to do some tweaks for ports to be used by nfs server.
https://medium.com/#aronasorman/creating-an-nfs-server-within-kubernetes-e6d4d542bbb9