In the "Juju" installation of kubernetes in Vsphere, we create pvc as follows,
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: db-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast
resources:
requests:
storage: 1Gi
with the storageClassName as "fast". What it the storage class we need to create a "PersistentVolumeClaim" in "Docker for windows" installation.
A StorageClass provides a way for administrators to describe the
“classes” of storage they offer. Different classes might map to
quality-of-service levels, or to backup policies, or to arbitrary
policies determined by the cluster administrators. Kubernetes itself
is unopinionated about what classes represent. This concept is
sometimes called “profiles” in other storage systems.
You can create several StorageClasses that fit your needs referring to vSphere examples in the official documentation:
vSphere
Create a StorageClass with a user specified disk format.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
diskformat: zeroedthick
diskformat: thin, zeroedthick and eagerzeroedthick. Default: "thin".
Create a StorageClass with a disk format on a user specified
datastore.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
diskformat: zeroedthick
datastore: VSANDatastore
datastore: The user can also specify the datastore in the StorageClass. The volume will be created on the datastore specified in
the storage class, which in this case is VSANDatastore. This field is
optional. If the datastore is not specified, then the volume will be
created on the datastore specified in the vSphere config file used to
initialize the vSphere Cloud Provider.
Storage Policy Management inside kubernetes
Using existing vCenter SPBM policy
One of the most important features of vSphere for Storage Management
is policy based Management. Storage Policy Based Management (SPBM) is
a storage policy framework that provides a single unified control
plane across a broad range of data services and storage solutions.
SPBM enables vSphere administrators to overcome upfront storage
provisioning challenges, such as capacity planning, differentiated
service levels and managing capacity headroom.
The SPBM policies can be specified in the StorageClass using the
storagePolicyName parameter.
Virtual SAN policy support inside Kubernetes
Vsphere Infrastructure (VI) Admins will have the ability to specify
custom Virtual SAN Storage Capabilities during dynamic volume
provisioning. You can now define storage requirements, such as
performance and availability, in the form of storage capabilities
during dynamic volume provisioning. The storage capability
requirements are converted into a Virtual SAN policy which are then
pushed down to the Virtual SAN layer when a persistent volume (virtual
disk) is being created. The virtual disk is distributed across the
Virtual SAN datastore to meet the requirements.
You can see Storage Policy Based Management for dynamic provisioning
of volumes for more details on how to use storage policies for
persistent volumes management.
There are few vSphere examples which you try out for persistent
volume management inside Kubernetes for vSphere.
Hope I found the answer,
kubectl get storageclass gives output as follows,
NAME PROVISIONER AGE
hostpath (default) docker.io/hostpath 22h
then, we can use 'hostpath' as the value for 'storageClassName'
Related
I am currently following this guide here to create a PV using an existing Azure File Share: https://learn.microsoft.com/en-us/azure/aks/azure-files-volume
The method is to store the storage account name and access key in a secret azure secret then use it in the csi section of the yaml file as below.
apiVersion: v1
kind: PersistentVolume
metadata:
name: azurefile
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: azurefile-csi
csi:
driver: file.csi.azure.com
readOnly: false
volumeHandle: unique-volumeid # make sure this volumeid is unique in the cluster
volumeAttributes:
resourceGroup: EXISTING_RESOURCE_GROUP_NAME # optional, only set this when storage account is not in the same resource group as agent node
shareName: aksshare
nodeStageSecretRef:
name: azure-secret
namespace: default
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- nosharesock
- nobrl
However, due to technical risk and security reasons, now I do not want to put storage account access key in the kubernetes namespace. Instead, I want to fetch the access key from Azure key vault and use it to mount the persistent volume to the azure files.
I have done some research and testing, but to no avail. Would appreciate help on this, thanks!
We can fetch the access key from keyvault to mount the persistent volume to fileshare using providerClass instead of storage account access key
I have created RG and AKS Cluster
While creating the AKS Cluster we have to enable the CSI Drivers
I have created the keyvault to secure our secrets
Resources>Keyvault>Create>RG,kvName>Review&Create
I have created the secrets using keyvaults
In KV Go-To secrets>click on Generate/import >give the name and secret value to create the secrets and its value(password)
Verify that your virtual machine scale set have their own system-assigned identity, if not we have to enable it
I have given the Access policy permissions to read the keyvault and its content
Go-To>keyvault>access-policy>create>
Permissions>select>secret Permissions
principal>select>id
application>select application id>create
Created the secretprovider class by using this provider class by this class will get the secrets from the key vault
secretproviderclass(check the file here)
Apply the below command to deploy the secret class
kubectl apply -f filename.yaml
By deploying the provider class the secrets will not be created, for that we have to create the POD which will mount the volume by utilizing the CSI drivers
pod.yaml(check this link)
Deploy the pod using below command
kubectl apply -f file.yaml
After the pod starts, the mounted content at the volume path that we specified in YAML file will be available
kubectl exec <pod_name> -- ls /mnt/secrets-store/
kubectl exec <pod_name> -- cat /mnt/secrets-store/<secret_name>
By using the above commands we will get the secret and secret value.
We have one bare metal machine with one SSD and one HDD. The pods in Minikube (k8s) will claim some pvc and get some volumes. We want to enforce that these volumes are actually on our SSD, not HDD. How to do that? Thanks very much!
p.s. What I have tried: When wanting a PVC, we find that Minikube will assign one on /tmp/hostpath-provisioner/.... In addition, IMHO this is the path inside the Docker that Minikube itself runs, not the host machine path. Thus, I have tried to minikube mount /data/minikube-my-tmp-hostpath-provisioner:/tmp/hostpath-provisioner where the /data of host bare metal machine is on SSD (not HDD). However, this makes the pods unhappy, and after a restart they all fail... In addition, I find that only new files will be written to the newly mounted path, while the existing files will still be inside the container...
This sounds exactly like the reason for which storage classes exist:
A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is unopinionated about what classes represent. This concept is sometimes called “profiles” in other storage systems.
So in other words - you can create multiple storage classes with different performance or other characteristics. And then decide which one is most appropriate for each claim they create.
For example, this is a storage class you can use on minikube:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
type: pd-ssd
And you'll probably also need to create a PV, you can do that using:
apiVersion: v1
kind: PersistentVolume
metadata:
name: some-name-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /tmp/path
Then, finally, the PVC would like this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: some-pvc
spec:
storageClassName: fast
resources:
requests:
storage: 100Mi
accessModes:
- ReadWriteOnce
I have a Jenkins pipeline using the kubernetes plugin to run a docker in docker container and build images:
pipeline {
agent {
kubernetes {
label 'kind'
defaultContainer 'jnlp'
yaml """
apiVersion: v1
kind: Pod
metadata:
labels:
name: dind
...
I also have a pool of persistent volumes in the jenkins namespace each labelled app=dind. I want one of these volumes to be picked for each pipeline run and used as /var/lib/docker in my dind container in order to cache any image pulls on each run. I want to have a pool and caches, not just a single one, as I want multiple pipeline runs to be able to happen at the same time. How can I configure this?
This can be achieved natively in kubernetes by creating a persistent volume claim as follows:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dind
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
selector:
matchLabels:
app: dind
and mounting it into the Pod, but I'm not sure how to configure the pipeline to create and cleanup such a persistent volume claim.
First of all, I think the way you think it can be achieved natively in kubernetes - wouldn't work. You either have to re-use same PVC which will make build pods to access same PV concurrently, or if you want to have a PV per build - your PVs will be stuck in Released status and not automatically available for new PVCs.
There is more details and discussion available here https://issues.jenkins.io/browse/JENKINS-42422.
It so happens that I wrote two simple controllers - automatic PV releaser (that would find and make Released PVs Available again for new PVCs) and dynamic PVC provisioner (for Jenkins Kubernetes plugin specifically - so you can define a PVC as annotation on a Pod). Check it out here https://github.com/plumber-cd/kubernetes-dynamic-reclaimable-pvc-controllers. There is a full Jenkinsfile example here https://github.com/plumber-cd/kubernetes-dynamic-reclaimable-pvc-controllers/tree/main/examples/jenkins-kubernetes-plugin-with-build-cache.
I have multi node kubernetes setup. I am trying to allocate a Persistent volume dynamically using storage classes with NFS volume plugin.
I found storage classes examples for glusterfs, aws-ebs, etc.but, I didn't find any example for NFS.
If I create PV and PVC only then NFS works very well(Without storage class).
I tried to write storage class file for NFS, by referring other plugins. please refer it below,
nfs-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
namespace: kube-system
name: my-storage
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
labels:
kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/nfs
parameters:
path: /nfsfileshare
server: <nfs-server-ip>
nfs-pv-claim.yaml
apiVersion: v1
metadata:
name: demo-claim
annotations:
volume.beta.kubernetes.io/storage-class: my-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
It didn't worked. So, my question is, Can we write a storage class for NFS? Does it support dynamic provisioing?
As of August 2020, here's how things look for NFS persistence on Kubernetes:
You can
Put an NFS volume on a Pod directly:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
nfs:
path: /foo/bar
server: wherever.dns
Manually create a Persistent Volume backed by NFS, and mount it with a Persistent Volume Claim (PV spec shown below):
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0003
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: slow
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /tmp
server: 172.17.0.2
Use the (now deprecated) NFS PV provisioner from external-storage. This was last updated two years ago, and has been officially EOL'd, so good luck. With this route, you can make a Storage Class such as the one below to fulfill PVCs from your NFS server.
Update: There is a new incarnation of this provisioner as kubernetes-sigs/nfs-subdir-external-provisioner! It seems to work in a similar way to the old nfs-client provisioner, but is much more up-to-date. Huzzah!
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: example-nfs
provisioner: example.com/nfs
mountOptions:
- vers=4.1
Evidently, CSI is the future, and there is a NFS CSI driver. However, it doesn't support dynamic provisioning yet, so it's not really terribly useful.
Update (December 2020): Dynamic provisioning is apparently in the works (on master, but not released) for the CSI driver.
You might be able to replace external-storage's NFS provisioner with something from the community, or something you write. In researching this problem, I stumbled on a provisioner written by someone on GitHub, for example. Whether such provisioners perform well, are secure, or work at all is beyond me, but they do exist.
I'm looking into doing the same thing. I found https://github.com/kubernetes-incubator/external-storage/tree/master/nfs, which I think you based your provisioner on?
I think an nfs provider would need to create a unique directory under the path defined. I'm not really sure how this could be done.
Maybe this is better of as an github issue on the kubernetes repo.
Dynamic storage provisioning using NFS doesn't work, better use glusterfs. There's a good tutorial with fixed to common problems while setting up.
http://blog.lwolf.org/post/how-i-deployed-glusterfs-cluster-to-kubernetes/
I also tried to enable the NFS provisioner on my kubernetes cluster and at first it didn't work, because the quickstart guide does not mention that you need to apply the rbac.yaml as well (I opened a PR to fix this).
The nfs provisioner works fine for me if I follow these steps on my cluster:
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs#quickstart
$ kubectl create -f deploy/kubernetes/deployment.yaml
$ kubectl create -f deploy/kubernetes/rbac.yaml
$ kubectl create -f deploy/kubernetes/class.yaml
Then you should be able to create PVCs like this:
$ kubectl create -f deploy/kubernetes/claim.yaml
You might want to change the folders used for the volume mounts in deployment.yaml to match it with your cluster.
The purpose of StorageClass is to create storage, e.g. from cloud providers (or "Provisioner" as they call it in the kubernetes docs). In case of NFS you only want to get access to existing storage and there is no creation involved. Thus you don't need a StorageClass. Please refer to this blog.
If you are using AWS, I believe you can use this image to create a NFS server:
https://hub.docker.com/r/alphayax/docker-volume-nfs
I'm setting up Jenkins on kubernetes. My current deployment looks like this:
spec:
containers:
- name: jenkins-master
image: jenkins:1.0
ports:
- containerPort: 8080
name: http
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
volumes:
- name: jenkins-home
emptyDir: {}
It works fine, but the data disappears if the pod gets destroyed, because the folder that jenkins uses to store data is mounted as an emptyDir, which means the data lives as long as the pod lives.
This is obviously not an optimal scenario, since the pod can be destroyed for many reasons, even during normal operations. I know I have to replace the emptyDir with something else, however, I'm not sure what should I use. I could provision a GCE disk, however, if you provision a disk under 200GB, you get a warning that your disk will be throttled. Since GCE disks can only by claimed by one writer at the time, it seems overkill, we would end up with an expensive bill if I used a GCE disk for every service that needs to hold persistent data.
To generalize the question:
You run several services on kubernetes, that need to hold a small amount of persistent, on-disk, data. What kind of storage do you provision for them?
What you're looking for is Persistent Volumes which essentially model an underlying persistent volume (say, AWS EBS or OpenStack Cinder volume) as resource.
This resource, named simply PersistentVolume defines the specification for the volume including e.g. its size and name. To actually use it you need to add PersistentVolumeClaim to your deployment yml which defines what your deployment expects from the persistent volume you want to attach to it - it is possible to have multiple matching persistent volumes for each claim just as it is possible that there won't be any volumes left to claim which is why this distinction exists.