I am seeing a significant performance difference between RWX (ReadWriteMany) and RWO (ReadWriteOnce) types of volumes.
Here is the yaml that creates these two volumes
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-rwx
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: longhorn
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-rwo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn
Here is a job/pod that mounts these two volumes inside it. let's say the rwx volume is mounted at /volume-rwx, likewise, the rwo volume is mounted at /volume-rwo
Here is the yaml for the job
apiVersion: batch/v1
kind: Job
metadata:
name: test
spec:
backoffLimit: 1
template:
spec:
containers:
- args:
- |
tail -f /dev/null
command:
- sh
- -c
image: alpine:3.16
name: test
volumeMounts:
- mountPath: /volume-rwx
name: volume-rwx
- mountPath: /volume-rwo
name: volume-rwo
restartPolicy: Never
volumes:
- name: volume-rwx
persistentVolumeClaim:
claimName: pvc-rwx
- name: volume-rwo
persistentVolumeClaim:
claimName: pvc-rwo
For testing, I will be at each mounted location inside the test pod. later I will download Virtual-Box source (not important) and extract it.
Here is the result when the extract extract performed from /volume-rwo
/volume-rwo # wget https://download.virtualbox.org/virtualbox/6.1.38/VirtualBox-6.1.38.tar.bz2
Connecting to download.virtualbox.org (23.201.200.86:443)
saving to 'VirtualBox-6.1.38.tar.bz2'
VirtualBox-6.1.38.ta 100% |************************************************************************************************************************************| 158M 0:00:00 ETA
'VirtualBox-6.1.38.tar.bz2' saved
/volume-rwo # time -p tar -xf VirtualBox-6.1.38.tar.bz2
real 49.29
user 43.55
sys 5.74
/volume-rwo #
```on
Also, this is the result when the same task is performed at /volume-rwx
```sh
/volume-rwx # wget https://download.virtualbox.org/virtualbox/6.1.38/VirtualBox-6.1.38.tar.bz2
Connecting to download.virtualbox.org (23.221.32.85:443)
saving to 'VirtualBox-6.1.38.tar.bz2'
VirtualBox-6.1.38.ta 100% |************************************************************************************************************************************| 158M 0:00:00 ETA
'VirtualBox-6.1.38.tar.bz2' saved
/volume-rwx # time -p tar -xf VirtualBox-6.1.38.tar.bz2
real 574.35
user 56.88
sys 44.23
/volume-rwx #
evidently, rwx took 10 times as much time to perform the same extraction.
I have checked and found this issue was fixed at a very early stage.
Whereas I am using Longhorn (v1.3.1).
Please help me solve it.
Thank you
Related
We have a docker image that is processing some files on a samba share.
For this we created a cifs share which is mounted to /mnt/dfs and files can be accessed in the container with:
docker run -v /mnt/dfs/project1:/workspace image
Now what I was aked to do is get the container into k8s and to acces a cifs share from a pod a cifs Volume driver usiong FlexVolume can be used. That's where some questions pop up.
I installed this repo as a daemonset
https://k8scifsvol.juliohm.com.br/
and it's up and running.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cifs-volumedriver-installer
spec:
selector:
matchLabels:
app: cifs-volumedriver-installer
template:
metadata:
name: cifs-volumedriver-installer
labels:
app: cifs-volumedriver-installer
spec:
containers:
- image: juliohm/kubernetes-cifs-volumedriver-installer:2.4
name: flex-deploy
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- mountPath: /flexmnt
name: flexvolume-mount
volumes:
- name: flexvolume-mount
hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
Next thing to do is add a PeristentVolume, but that needs a capacity, 1Gi in the example. Does this mean that we lose all data on the smb server? Why should there be a capacity for an already existing server?
Also, how can we access a subdirectory of the mount /mnt/dfs from within the pod? So how to access data from /mnt/dfs/project1 in the pod?
Do we even need a PV? Could the pod just read from the host's mounted share?
apiVersion: v1
kind: PersistentVolume
metadata:
name: mycifspv
spec:
capacity:
storage: 1Gi
flexVolume:
driver: juliohm/cifs
options:
opts: sec=ntlm,uid=1000
server: my-cifs-host
share: /MySharedDirectory
secretRef:
name: my-secret
accessModes:
- ReadWriteMany
No, that field has no effect on the FlexVol plugin you linked. It doesn't even bother parsing out the size you pass in :)
Managed to get it working with the fstab/cifs plugin.
Copy its cifs script to /usr/libexec/kubernetes/kubelet-plugins/volume/exec and give it execute permissions. Also restart kubelet on all nodes.
https://github.com/fstab/cifs
Then added
containers:
- name: pablo
image: "10.203.32.80:5000/pablo"
volumeMounts:
- name: dfs
mountPath: /data
volumes:
- name: dfs
flexVolume:
driver: "fstab/cifs"
fsType: "cifs"
secretRef:
name: "cifs-secret"
options:
networkPath: "//dfs/dir"
mountOptions: "dir_mode=0755,file_mode=0644,noperm"
Now there is the /data mount inside the container pointing to //dfs/dir
I want to copy a text file to a pod on minikube. But I get the timeout error.
scp -r /Users/joe/Downloads/Archive/data.txt docker#192.168.49.2:/home/docker
I got the ip address (192.168.49.2) with:
minikube ip
Eventually I would like that the file appear on the persistentVolumeClaim/persistentVolume (that will be great!!)
The yaml for the PersistentVolume is:
kind: PersistentVolume
apiVersion: v1
metadata:
name: my-pv
spec:
storageClassName: local-storage
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
The yaml for the PersistentVolumeClaim is:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Mi
The yaml for the pod is:
kind: Pod
apiVersion: v1
metadata:
name: my-pvc-pod
spec:
containers:
- name: busybox
image: busybox
command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
volumeMounts:
- mountPath: "/mnt/storage"
name: my-storage
volumes:
- name: my-storage
persistentVolumeClaim:
claimName: my-pvc
Eventually I would like that the file appear on the persistentVolumeClaim/persistentVolume.
You can achieve that with mounting the host directory into the guest using minikube mount command:
minikube mount <source directory>:<target directory>
Whereas the the <source directory> is the host directory and <target directory> is the guest/minikube directory.
And then use that <target directory> and create pv with hostPath:
apiVersion: v1
kind: PersistentVolume
metadata:
name: task-pv-volume
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "<target-directory"
Depending also driver, some of them have built-in host folder sharing. You can check them here.
If you need to mount only part of the volume, in your case a single file, you can use subPath to specify the part that must be mounted. This answer explains it well.
I am a bit confused here. It does work as normal docker container but when it goes inside a pod it doesnt. So here is how i do it.
Dockerfile in my local to create the image and publish to docker registry
FROM alpine:3.7
COPY . /var/www/html
CMD tail -f /dev/null
Now if i just pull the image(after deleting the local) and run as a container. It works and i can see my files inside /var/www/html.
Now i want to use that inside my kubernetes cluster.
Def : Minikube --vm-driver=none
I am running kube inside minikube with driver none option. So for single node cluster.
EDIT
I can see my data inside /var/www/html if i remove volume mounts and claim from deployment file.
Deployment file
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
io.kompose.service: app
name: app
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: app
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
containers:
- image: kingshukdeb/mycode
name: pd-mycode
resources: {}
volumeMounts:
- mountPath: /var/www/html
name: claim-app-storage
restartPolicy: Always
volumes:
- name: claim-app-storage
persistentVolumeClaim:
claimName: claim-app-nginx
status: {}
PVC file
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
io.kompose.service: app-nginx1
name: claim-app-nginx
spec:
storageClassName: testmanual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
status: {}
PV file
apiVersion: v1
kind: PersistentVolume
metadata:
name: app-nginx1
labels:
type: local
spec:
storageClassName: testmanual
capacity:
storage: 100Mi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/volumes/app"
Now when i run these files it creates the pod, pv, pvc and pvc is bound to pv. But if i go inside my container i dont see my files. hostpath is /data/volumes/app . Any ideas will be appreciated.
When PVC is bound to a pod, volume is mounted in location described in pod/deployment yaml file. In you case: mountPath: /var/www/html. That's why files "baked into" container image are not accessible (simple explanation why here)
You can confirm this by exec to the container by running kubectl exec YOUR_POD -i -t -- /bin/sh, and running mount | grep "/var/www/html".
Solution
You may solve this in many ways. It's best practice to keep your static data separate (i.e. in PV), and keep the container image as small and fast as possible.
If you transfer files you want to mount in PV to your hosts path /data/volumes/app they will be accessible in your pod, then you can create new image omitting the COPY operation. This way even if pod crashes changes to files made by your app will be saved.
If PV will be claimed by more than one pod, you need to change accessModes as described here:
The access modes are:
ReadWriteOnce – the volume can be mounted as read-write by a single node
ReadOnlyMany – the volume can be mounted read-only by many nodes
ReadWriteMany – the volume can be mounted as read-write by many nodes
In-depth explanation of Volumes in Kubernetes docs: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
I'm working in Kubernetes in GCP and I'm having problems with volumes and persistent disks.
I'm using Directus 7 (CMS Headless), which saves most of its information in the database except the files that are uploaded, these files are in the /var/www/html/public/uploads folder (tested locally with docker-compose and works fine), and that folder is the one I'm trying to save on the persistent disk.
No error occurs but when restart the Kubernetes Pod i lose the uploaded images (they are not being saved on the disk).
This is my configuration:
apiVersion: v1
kind: PersistentVolume
metadata:
name: directus-pv
namespace: default
spec:
storageClassName: ""
capacity:
storage: 100G
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: directus-disk
fsType: ext4
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: directus-pvc
namespace: default
labels:
app: .....
spec:
storageClassName: ""
volumeName: directus-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100G
And in the deploy.yaml:
volumeMounts:
- name: api-disk
mountPath: /var/www/html/public/uploads
readOnly: false
volumes:
- name: api-disk
persistentVolumeClaim:
claimName: directus-pvc
Thanks for the help
Remove namespace property from pv and pvc manifest. They are shared resources in the cluster.
Remove storage class property as well.
I presume that your manually provisioned persistence volume directus-pv, is being created somehow with PersistentVolumeReclaimPolicy=*Recycle. That's the only possible reason that could cause data erase on each POD restart.
I'm not able to reproduce your case with the provided manifest files,
but I tried the following test:
Create gcePersistentDisk
Create PersistentVolume
Create PersistentVolumeClaim
Create ReplicaSet (replicas=1) like this one
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: busybox-list-uploads
spec:
replicas: 1
template:
metadata:
labels:
app: busybox-list-uploads
version: "2"
spec:
containers:
- image: busybox
args: [/bin/sh, -c, 'sleep 9999' ]
volumeMounts:
- mountPath: /var/www/html/public/uploads
name: api-disk
name: busybox
volumes:
- name: api-disk
persistentVolumeClaim:
claimName: directus-pvc
Write some file into mounted folder /var/www/html/public/uploads
Restart POD (=kill the POD) by resizing replica to 0 then to 1
List content of /var/www/html/public/uploads on newly created POD
for i in busybox-list-uploads-dgfbc; do kubectl exec -it $i -- ls /var/www/html/public/uploads; done;
lost+found picture_from_busybox-list-uploads-ng4t6.png
As you can see output shows clearly, that data survives POD restart
* you can verify it with cmd: kubectl get pv/directus-pv -o yaml
I want to be able to mount an unknown number of config files in /etc/configs
I have added some files to the configmap using:
kubectl create configmap etc-configs --from-file=/tmp/etc-config
The number of files and file names are never going to be known and I would like to recreate the configmap and the folder in the Kubernetes container should be updated after sync interval.
I have tried to mount this but I'm not able to do so, the folder is always empty but I have data in the configmap.
bofh$ kubectl describe configmap etc-configs
Name: etc-configs
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
file1.conf:
----
{
... trunkated ...
}
file2.conf:
----
{
... trunkated ...
}
file3.conf:
----
{
... trunkated ...
}
Events: <none>
I'm using this one in the container volumeMounts:
- name: etc-configs
mountPath: /etc/configs
And this is the volumes:
- name: etc-configs
configMap:
name: etc-configs
I can mount individual items but not an entire directory.
Any suggestions about how to solve this?
You can mount the ConfigMap as a special volume into your container.
In this case, the mount folder will show each of the keys as a file in the mount folder and the files will have the map values as content.
From the Kubernetes documentation:
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
...
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name: special-config
I'm feeling really stupid now.
Sorry, My fault.
The Docker container did not start so I was manually staring it using docker run -it --entrypoint='/bin/bash' and I could not see any files from the configMap.
This does not work since docker don't know anything about my deployment until Kubernetes starts it.
The docker image was failing and the Kubernetes config was correct all the time.
I was debugging it wrong.
With your config, you're going to mount each file listed in your configmap.
If you need to mount all file in a folder, you shouldn't use configmap, but a persistenceVolume and persistenceVolumeClaims:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-volume-jenkins
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/pv-jenkins"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pv-claim-jenkins
spec:
accessModes:
- ReadWriteOnce
storageClassName: ""
resources:
requests:
storage: 50Gi
In your deployment.yml:
volumeMounts:
- name: jenkins-persistent-storage
mountPath: /data
volumes:
- name: jenkins-persistent-storage
persistentVolumeClaim:
claimName: pv-claim-jenkins
You can also use the following:
kubectl create configmap my-config --from-file=/etc/configs
to create the config map with all files in that folder.
Hope this helps.