I have setup kubernetes Cluster and mounted volume mount as gcePersistentDisk in Google Cloud, It claims and mount successfully in Pods.
But i want to access this volume externally so that i can write it through git/ssh or manual. As disk is Already used and mounted i cannot access it.
How to write files through externally?
gcePersistentDisk is a network-based disk, and provisioned volumes can only be used by GCE
instances in the same project and zone.
The fact is that this kind of resource supports readWriteOnce and ReadOnlyMany.
You can use a GCE persistent storage to share data as read-only between multiple pods in the
same zone.
Back to your question: you can write on this volume only from one pod. No other pods can
use it as write storage - neither external nor from the same project.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: php
labels:
app: php
spec:
replicas: 1
selector:
matchLabels:
app: php
template:
metadata:
labels:
app: php
spec:
containers:
- image: php:7.1-apache
imagePullPolicy: Always
name: php
resources:
requests:
cpu: 200m
ports:
- containerPort: 80
name: php
volumeMounts:
- name: php-persistent-storage
mountPath: /var/www
volumes:
- name: php-persistent-storage
gcePersistentDisk:
pdName: php-phantomjs-disk
fsType: ext4
Related
We have a docker image that is processing some files on a samba share.
For this we created a cifs share which is mounted to /mnt/dfs and files can be accessed in the container with:
docker run -v /mnt/dfs/project1:/workspace image
Now what I was aked to do is get the container into k8s and to acces a cifs share from a pod a cifs Volume driver usiong FlexVolume can be used. That's where some questions pop up.
I installed this repo as a daemonset
https://k8scifsvol.juliohm.com.br/
and it's up and running.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cifs-volumedriver-installer
spec:
selector:
matchLabels:
app: cifs-volumedriver-installer
template:
metadata:
name: cifs-volumedriver-installer
labels:
app: cifs-volumedriver-installer
spec:
containers:
- image: juliohm/kubernetes-cifs-volumedriver-installer:2.4
name: flex-deploy
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- mountPath: /flexmnt
name: flexvolume-mount
volumes:
- name: flexvolume-mount
hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
Next thing to do is add a PeristentVolume, but that needs a capacity, 1Gi in the example. Does this mean that we lose all data on the smb server? Why should there be a capacity for an already existing server?
Also, how can we access a subdirectory of the mount /mnt/dfs from within the pod? So how to access data from /mnt/dfs/project1 in the pod?
Do we even need a PV? Could the pod just read from the host's mounted share?
apiVersion: v1
kind: PersistentVolume
metadata:
name: mycifspv
spec:
capacity:
storage: 1Gi
flexVolume:
driver: juliohm/cifs
options:
opts: sec=ntlm,uid=1000
server: my-cifs-host
share: /MySharedDirectory
secretRef:
name: my-secret
accessModes:
- ReadWriteMany
No, that field has no effect on the FlexVol plugin you linked. It doesn't even bother parsing out the size you pass in :)
Managed to get it working with the fstab/cifs plugin.
Copy its cifs script to /usr/libexec/kubernetes/kubelet-plugins/volume/exec and give it execute permissions. Also restart kubelet on all nodes.
https://github.com/fstab/cifs
Then added
containers:
- name: pablo
image: "10.203.32.80:5000/pablo"
volumeMounts:
- name: dfs
mountPath: /data
volumes:
- name: dfs
flexVolume:
driver: "fstab/cifs"
fsType: "cifs"
secretRef:
name: "cifs-secret"
options:
networkPath: "//dfs/dir"
mountOptions: "dir_mode=0755,file_mode=0644,noperm"
Now there is the /data mount inside the container pointing to //dfs/dir
docker engine supports data volumes
A Docker data volume persists after a container is deleted
docker run and docker-compose both support it:
docker run --volume data_vol:/mount/point
docker-compose with named volumes using top-level volumes key
kubernetes also supports persistent volumes, but does it support the same concept of having a data volume - that is, a volume which resides within a container?
if kubernetes supports a data volume (within a container):
would appreciate any reference to the documentation (or an example)
does it also support the migration of the data volume in the same manner it supports the migration of regular containers?
i found some related questions, but couldn't get the answer i am looking for.
What you are trying to say is:
If you do not specify a host path for a docker volume mount, docker dynamically provisions a path and persist it between restarts.
"that is, a volume which resides within a container"
Volume is generated outside of container and mounted later.
For example:
# data_vol location is decided by docker installation
docker run --volume data_vol:/mount/point
# host path is explicitly given
docker run --volume /my/host/path:/mount/point
In kubernetes terms, this is similar to dynamic provisioning. If you want dynamic provisioning, you need to have Storage classes depending on your storage backend.
Please read https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/ .
If you want to specify a host path, following is an example. You can also achieve similar results by using NFS, block storage etc.
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
hostPath:
path: /home/user/my-vol
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-ss
spec:
replicas: 1
selector:
matchLabels:
app: my-ss
serviceName: my-svc
template:
metadata:
labels:
app: my-ss
spec:
containers:
- image: ubuntu
name: my-container
volumeMounts:
- mountPath: /my-vol
name: my-vol
volumeClaimTemplates:
- metadata:
name: my-vol
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
selector:
matchLabels:
app: my-ss
I am running a docker image that has certain configuration files within it. I need to persist/mount the same folder to the disk as new files will get added later on. When I use standard volume mount in kubernetes, it mounts an empty directory without the intial configuration files. How do I make sure my initial files are copied to the volume while mounting?
- mountPath: /tmp
name: my-vol
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: my-vol
persistentVolumeClaim:
claimName: wso2-disk2```
A possible solution could be the use the node storage mounted on containers (easiest way) or using a DFS solution like NFS, GlusterFS, and so on.
Another and recommended way to achieve what you need is to use a persistent volumes to share the same files between your containers.
Assuming you have a kubernetes cluster that has only one Node, and you want to share the path /mtn/data of your node with your pods (Source):
Create a PersistentVolume:
A hostPath PersistentVolume uses a file or directory on the Node to emulate network-attached storage.
apiVersion: v1
kind: PersistentVolume
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
Create a PersistentVolumeClaim:
Pods use PersistentVolumeClaims to request physical storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: task-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
Look at the PersistentVolumeClaim:
kubectl get pvc task-pv-claim
The output shows that the PersistentVolumeClaim is bound to your PersistentVolume, task-pv-volume.
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
task-pv-claim Bound task-pv-volume 10Gi RWO manual 30s
Create a deployment with 2 replicas for example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: task-pv-claim
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/mnt/data"
name: task-pv-storage
Now you can check inside both container the path /mnt/data has the same files.
If you have cluster with more than 1 node I recommend you to think about the other types of persistent volumes or using DFS.
References:
Configure persistent volumes
Persistent volumes
Volume Types
The suggested way to provide configurations to your pod is by creating a configmap for your configurations and mount it in your pod using volumes. This guide ( https://kubernetes.io/docs/concepts/storage/volumes/#configmap) descibes how to do that.
Other ways are to create a persistent volume and persistent volume claim in your cluster and copy your configuration file in that path. Mount the persistent volume in your pod.
You can also copy your configuration on one of the nodes in your cluster and mount that path using hostPath but this requires that your pod should also run on the same node as it tries to look for the path in that node. (Not a recommended approach)
Create configmap of the folder you would like to mount, the following creates configmap consisting of all the files in your-folder:
kubectl create configmap your-config --from-file=your-folder/
Then mount this to the volume and you will have the initial files in your folder. And note that you will need to mount it to subpath since you dont want it to overwrite everything in the directory.
I have directly mounted NFS volume for mysql data, need to implement storage policy for retaining data across pod deletion, and to avoid any corruption. please recommend some useful.
I did not find a way to enable persistentVolumeReclaimPolicy: Retain in directly mounted volumes . I know it can be done from PV/PVC creation but is it possible from statefulset volumes... Some guidelines is needed in understanding the yaml options for a particular object, where to get all the options(parameters) available for an object. currently googling for each options and trying - so hard.
I could not mount a configmap file (my.cnf) to a file in the pod. it removes the underlying files in the mountpath. curious to know how it is handled generally, do we need separate mount path for each config file.
code block
apiVersion: v1
kind: Service
metadata:
name: mymariadb
labels:
app: mymariadb
spec:
ports:
- port: 3306
name: mysql
targetPort: mysql
nodePort: 30003
type: NodePort
selector:
app: mymariadb
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mymariadb
labels:
app: mymariadb
spec:
serviceName: "mymariadb"
selector:
matchLabels:
app: mymariadb
template:
metadata:
labels:
app: mymariadb
spec:
containers:
- name: mariadb
image: mariadb:10.3.7
env:
- name: MYSQL_ROOT_PASSWORD
value: xxxx
ports:
- name: mysql
containerPort: 3306
volumeMounts:
- name: data
mountPath: /data
subPath: mysql
- name: conf
mountPath: /etc/mysql # /conf.d removing files
resources:
requests:
cpu: 500m
memory: 2Gi
volumes:
- name: data
nfs:
server: 10.12.32.41
path: /data/mymariadb
spec:
persistentVolumeReclaimPolicy: Retain # not taking
- name: conf
configMap:
name: mycustconf
items:
- key: my.cnf
path: my.cnf
Firstly, I did not suggest nfs mount in Kubernetes platform for two reasons. From security perspective, another container can access the nfs mount on the worker nodes. The Second, from performances perspective, the connection between worker nodes and storage will be slower, to compare to another solutions. As you know, performance is so critical for db connections. I think you should evaluate that.
I suggest to you use one of the Cloud Native Storages. You can view them in the link below. Ceph and Gluster are popular products.
https://landscape.cncf.io/category=cloud-native-storage&format=card-mode&grouping=category
If you really want to continue with the nfs solution, you can check two points:
1) Did you check the access list on the storage appliance? You should see the worker nodes for the nfs mount.
2) After you try to mount the nfs storage on the worker nodes, you can try to import the deployment on your kubernetes cluster.
I'm trying to setup Jenkins to run in a container on Kubernetes, but I'm having trouble persisting the volume for the Jenkins home directory.
Here's my deployment.yml file. The image is based off jenkins/jenkins
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins-deployment
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: 1234567.dkr.ecr.us-east-1.amazonaws.com/mycompany/jenkins
imagePullPolicy: "Always"
ports:
- containerPort: 8080
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-home
emptyDir: {}
However, if i then push a new container to my image repository and update the pods using the below commands, Jenkins comes back online but asks me to start from scratch (enter admin password, none of my Jenkins jobs are there, no plugins etc)
kubectl apply -f kubernetes (where my manifests are stored)
kubectl set image deployment/jenkins-deployment jenkins=1234567.dkr.ecr.us-east-1.amazonaws.com/mycompany/jenkins:$VERSION
Am I misunderstanding how this volume mount is meant to work?
As an aside, I also have backup and restore scripts which backup the Jenkins home directory to s3, and download it again, but that's somewhat outside the scope of this issue.
You should use PersistentVolumes along with StatefulSet instead of Deployment resource if you wish your data to survive re-deployments|restarts of your pod.
You have specified the volume type EmptyDir. This will essentially mount an empty directory on the kube node that runs your pod. Every time you restart your deployment, the pod could move between kube hosts and the empty dir isn't present, so your data isn't persisting across restarts.
I see you're pulling you image from an ECR repository, so I'm assuming you're running k8s in AWS.
You'll need to configure a StorageClass for AWS. If you've provisioned k8s using something like kops, this will already be configured. You can confirm this by doing kubectl get storageclass - the provisioner should be configured as EBS:
NAME PROVISIONER
gp2 (default) kubernetes.io/aws-ebs
Then, you need to specify a persistentvolumeclaim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp2 # must match your storageclass from above
resources:
requests:
storage: 30Gi
You can now the pv claim on your deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins-deployment
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: 1234567.dkr.ecr.us-east-1.amazonaws.com/mycompany/jenkins
imagePullPolicy: "Always"
ports:
- containerPort: 8080
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
persistentVolumeClaim:
claimName: jenkins-data # must match the claim name from above