I'm looking for a pattern that allows to share volumes between two containers running on the same pod in Kubernetes.
My use case is:
I have a Ruby on Rails application running inside a docker container.
The docker image contains static assets in /app/<app-name>/public directory, and I need to access those assets from the nginx container running alongside in the same pod.
In 'vanilla' docker I would have used --volumes-from flag to share this directory:
docker run --name app -v /app/<app-dir>/public <app-image>
docker run --volumes-from app nginx
After reading this doc: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/volumes.md
I tried this (only relevant entries presented):
spec:
containers:
- image: <app-image>
name: <app-name>
volumeMounts:
- mountPath: /app/<app-name>/public
name: assets
- image: nginx
name: nginx
volumeMounts:
- mountPath: /var/www/html
name: assets
readOnly: true
volumes:
- name: assets
hostPath:
path: /tmp/assets
But:
Even though /tmp/assets on the node exists, it's empty
/app/<app-name>/public inside the app container is also empty
As a workaround I'm gonna try to populate the shared directory when the application container is up (simply cp /app/<app-name>/public/* to shared directory), but I really dislike this idea.
Question: how to mimic --volumes-from in Kubernetes, or if there is no direct counterpart, how can I share files from one container to other running in the same pod ?
apiVersion: v1beta3
Client Version: version.Info{Major:"0", Minor:"17", GitVersion:"v0.17.0", GitCommit:"82f8bdac06ddfacf493a9ed0fedc85f5ea62ebd5", GitTreeState:"clean"}
Server Version: version.Info{Major:"0", Minor:"17", GitVersion:"v0.17.0", GitCommit:"82f8bdac06ddfacf493a9ed0fedc85f5ea62ebd5", GitTreeState:"clean"}
[update-2016-8] In latest Kubernetes release, you can use a very nice feature named init-container to replace the postStart part in my answer below, which will make sure the container order.
apiVersion: v1
kind: Pod
metadata:
name: javaweb-2
spec:
initContainers:
- name: war
image: resouer/sample:v2
command: ["cp", "/sample.war", "/app"]
volumeMounts:
- mountPath: /app
name: app-volume
containers:
- name: tomcat
image: resouer/mytomcat:7.0
command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
volumeMounts:
- mountPath: /root/apache-tomcat-7.0.42-v2/webapps
name: app-volume
ports:
- containerPort: 8080
hostPort: 8001
volumes:
- name: app-volume
emptyDir: {}
NOTE: initContainer is still a beta feature so the work version of this yaml is actually like: http://kubernetes.io/docs/user-guide/production-pods/#handling-initialization, please notice the pod.beta.kubernetes.io/init-containers part.
---original answer begin---
Actually, you can. You need to use container life cycle handler to control what files/dirs you want to share with other containers. Like:
---
apiVersion: v1
kind: Pod
metadata:
name: server
spec:
restartPolicy: OnFailure
containers:
- image: resouer/sample:v2
name: war
lifecycle:
postStart:
exec:
command:
- "cp"
- "/sample.war"
- "/app"
volumeMounts:
- mountPath: /app
name: hostv1
- name: peer
image: busybox
command: ["tail", "-f", "/dev/null"]
volumeMounts:
- name: hostv2
mountPath: /app/sample.war
volumes:
- name: hostv1
hostPath:
path: /tmp
- name: hostv2
hostPath:
path: /tmp/sample.war
Please check my gist for more details:
https://gist.github.com/resouer/378bcdaef1d9601ed6aa
And of course you can use emptyDir. Thus, war container can share its /sample.war to peer container without mess peer's /app directory.
If we can tolerate /app been overridden, it will be much simpler:
---
apiVersion: v1
kind: Pod
metadata:
name: javaweb-2
spec:
restartPolicy: OnFailure
containers:
- image: resouer/sample:v2
name: war
lifecycle:
postStart:
exec:
command:
- "cp"
- "/sample.war"
- "/app"
volumeMounts:
- mountPath: /app
name: app-volume
- image: resouer/mytomcat:7.0
name: tomcat
command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
volumeMounts:
- mountPath: /root/apache-tomcat-7.0.42-v2/webapps
name: app-volume
ports:
- containerPort: 8080
hostPort: 8001
volumes:
- name: app-volume
emptyDir: {}
The answer is - for now - you can't. Here's a couple of discussion threads from the Kubernetes issues:
https://github.com/GoogleCloudPlatform/kubernetes/issues/6120
https://github.com/GoogleCloudPlatform/kubernetes/issues/831
However, may I suggest that you have an alternate design that might work better?
If your assets are locked at the point of the container going live,
you could use something like gitRepo
volume which would copy it to an emptyDir at the point of going live, and would mean you wouldn't have to move the content around at
all, just download it directly to the shared directory.
If your assets are locked at the point of the container
being built, it's probably best to copy them in at that point, using
the Docker COPY command.
If you really want to stick with the way you're doing it, you would have to copy the content to the emptyDir volume, which is designed for exactly what you're looking for (minus the lack of having to copy it in).
NFS[1] volumes also could solve your problem, but may be overly complex.
Additionally, I'd recommend that these two services exist in different pods, so you can scale each separately. You can create a service endpoint to communicate between them if you need to.
[1] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/nfs/nfs-web-pod.yaml
Further update from the future:
There is now a FlexVol plugin for Docker volumes: https://github.com/dims/docker-flexvol
At the time of writing, FlexVol is still an alpha feature though, so caveat emptor.
Kubernetes has its own volume types and these are most used volume type:
emptyDir
secret
gitRepo
hostPath (similar to --volumes-from)
config Maps
persistent storage (storage disks provided by cloud platforms)
You can find more about kubernets volumes here -https://kubernetes.io/docs/concepts/storage/volumes/
an example of hostpath volume :
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
hostpath will mount host/node directory to container directory.Multiple containers inside a pod can use different or same volumes.You need to mention it in each container.
hostPath volumes are independent of pod lifecycle but it create tight coupling between node and pod , you should avoid using hostPath.
If you are using Docker v17.0.5 or greater you can use a multi-stage build to copy files from one of your containers to the other during build time. This is a great primer on the advanced features at https://medium.com/#tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae
The way I used it to copy static assets from my backend container into Nginx proxy is
ARG API_BACKEND_CONTAINER="api:backend"
FROM $API_BACKEND_CONTAINER as source
FROM nginx:mainline-alpine
ARG NGINX_ROOT=/usr/share/nginx/html/
COPY --from=source /var/share/api/static/ ${NGINX_ROOT}
The great thing is that because the API_BACKEND_CONTAINER is a build arg I'm able to pass in the tag of the latest API build.
Related
I have containerized microservice built with Java. This application uses the default /config-volume directory when it searches for property files.
Previously I manually deployed via Dockerfile, and now I'm looking to automate this process with Kubernetes.
The container image starts the microservice immediately so I need to add properties to the config-volume folder immediately. I accomplished this in Docker with this simple Dockerfile:
FROM ########.amazon.ecr.url.us-north-1.amazonaws.com/company/image-name:1.0.0
RUN mkdir /config-volume
COPY path/to/my.properties /config-volume
I'm trying to replicate this type of behavior in a kubernetes deployment.yaml but I have found no way to do it.
I've tried performing a kubectl cp command immediately after applying the deployment and it sometimes works, but it can result in a race condition which cause the microservice to fail at startup.
(I've redacted unnecessary parts)
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-service
spec:
replicas: 1
template:
spec:
containers:
- env:
image: ########.amazon.ecr.url.us-north-1.amazonaws.com/company/image-name:1.0.0
name: my-service
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /config-volume
name: config-volume
volumes:
- name: config-volume
emptyDir: {}
status: {}
Is there a way to copy files into a volume inside the deployment.yaml?
You are trying to emulate a ConfigMap using volumes. Instead, put your configuration into a ConfigMap, and mount that to your deployments. The documentation is there:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
Once you have your configuration as a ConfigMap, mount it using something like this:
...
containers:
- name: mycontainer
volumeMounts:
- name: config-volume
mountPath: /config-volume
volumes:
- name: config-volume
configMap:
name: nameOfConfigMap
I'm using the kubernetes plugin to setup a pipeline on jenkins to compile some code.
MY GOAL:
In this pipeline, I'm trying to access some data from a docker container to use it as a cache in a second on (as shown below).
apiVersion: v1
kind: Pod
metadata:
name: cache-test
spec:
restartPolicy: Never
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: cache-container
image: cache:latest
volumeMounts:
- name: shared-data
mountPath: /cache
command:
- cat
- name: debian-container
image: debian
volumeMounts:
- name: shared-data
mountPath: /pod-data
command:
- cat
PROBLEM:
My issue is that when I mount my shared-folder in /cache directly, all my data get erased (overwritten).
WORK AROUND:
One work around would be to to create an intermediate directory where I can copy my data:
apiVersion: v1
kind: Pod
metadata:
name: cache-test
spec:
restartPolicy: Never
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: cache-container
image: cache:latest
volumeMounts:
- name: shared-data
mountPath: /shared-folder
command:
- cat
- name: debian-container
image: debian
volumeMounts:
- name: shared-data
mountPath: /pod-data
command:
- cat
And the in my Jenkins pipeline add this step:
container('cache-container') {
sh """#!/usr/bin/env bash
set -exu
cp -r /cache/* /shared-folder
"""
} // container
QUESTION:
Is there a way to avoid this copy step? Maybe a kubernetes volume setup that doesn't overwrite what's in the container?
I went through the documentation couple times without finding anything..
You can't use a container as a cache without copying data to a volume shared with other containers. But you probably shouldn't anyway.
You probably want to move your cache out of the container and make it a PersistentVolume. PersistentVolumeClaims can claim PV and then pods can mount these PVCs.
The problem with PVs, no matter static or dynamic they are - they won't be Available for PVCs when previously locking PVC was deleted. They will stuck in Released state. Kubernetes doesn't do it automatically for a reason - workloads aren't supposed to have access to data from other workloads. For when they do - the idiomatic Kubernetes way to do it is StatefulSets, so Kubernetes guarantees that only the replicas of the same workload may claim the old data. Unfortunately it's just doesn't work for a build caches.
I wrote two simple controllers - automatic PV releaser (that would find and make Released PVs Available again for new PVCs) and dynamic PVC provisioner (for Jenkins Kubernetes plugin specifically - so you can define a PVC as annotation on a Pod). Check it out here https://github.com/plumber-cd/kubernetes-dynamic-reclaimable-pvc-controllers. There is a full Jenkinsfile example here https://github.com/plumber-cd/kubernetes-dynamic-reclaimable-pvc-controllers/tree/main/examples/jenkins-kubernetes-plugin-with-build-cache.
I've dockerized a python project that requires the use of several CSVs (~2gb). In order to keep image size down I didn't include the CSVs in the build, instead opting to give the running container the data from a directory outside the container through a volume. Locally, when running through docker, I can just do
docker run -v ~/local/path/:/container/path my-image:latest
This works, but I'm not sure how to go about doing this in Kubernetes. I've been reading the documentation and am confused by the number of volume types, where the actual CSVs should be stored, etc.
Based on the information about the project that I've provided, is there an obvious solution?
If you'd like to replicate that exact behavior from Docker the most common way to do it is to use hostPath. Something like this:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: my-image:latest
name: my-container
volumeMounts:
- mountPath: /container/path
name: test-volume
volumes:
- name: test-volume
hostPath:
path: /usr/local/path
type: Directory
Here is a typical example of sharing between containers. You can keep your data in a separate container and code in a different container.
https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/
apiVersion: v1
kind: Pod
metadata:
name: two-containers
spec:
restartPolicy: Never
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: nginx-container
image: nginx
volumeMounts:
- name: shared-data
mountPath: /usr/share/nginx/html
- name: debian-container
image: debian
volumeMounts:
- name: shared-data
mountPath: /pod-data
command: ["/bin/sh"]
args: ["-c", "echo Hello from the debian container > /pod-data/index.html"]
Hope it helps.
I have a command to run docker,
docker run --name pre-core -itdp 8086:80 -v /opt/docker/datalook-pre-core:/usr/application app
In above command, /opt/docker/datalook-pre-core is host directory, /usr/application is container directory. The purpose is that container directory maps to host directory. So when container crashes, the directory functions as storage and data on it would be saved.
When I am going to use kubernetes to create a pod for this containter, how to write pod.yaml file?
I guess it is something like following:
apiVersion: v1
kind: Pod
metadata:
name: app-ykt
labels:
app: app-ykt
purpose: ykt_production
spec:
containers:
- name: app-ykt
image: app
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumnMounts:
- name: volumn-app-ykt
mountPath: /usr/application
volumns:
- name: volumn-app-ykt
？？？？
I do not know what's the exact properties in yaml I shall write in my case?
This would be a hostPath volume: https://kubernetes.io/docs/concepts/storage/volumes/
volumes:
- name: volumn-app-ykt
hostPath:
# directory location on host
path: /opt/docker/datalook-pre-core
# this field is optional
type: Directory
However remember that while a container crash won't move things, other events can cause a pod to move to a different host so you need to be prepared to both deal with cold caches and to clean up orphaned caches.
In docker-compose I was used to create volumes in this way:
volumes:
- ./server:/home/app
- /home/app/node_modules
in order to solve the problem of node_modules.
How could I approach the problem in kubernetes?
I've created the following config
spec:
containers:
- env: []
image: "image"
volumeMounts:
- mountPath: /home/app
name: vol-app
- mountPath: /home/app/node_modules
name: vol-node-modules
name: demo
volumes:
- name: vol-app
hostPath:
path: /Users/me/server
- name: vol-node-modules
emptyDir: {}
but it doesn't work. the node_modules is empty
I just transitioned from Docker Swarm to Kubernetes and had the same question. The solution I settled on makes use of init containers (https://kubernetes.io/docs/concepts/workloads/pods/init-containers/).
The general idea is that init containers allow you to copy the node_modules directory from your image into an empty volume, then mount that volume in your application's pod.
kind: Deployment
apiVersion: apps/v1
metadata:
...
spec:
...
template:
...
spec:
initContainers:
- name: frontend-clone
image: YOUR_REGISTRY/YOUR_IMAGE
command:
- cp
- -a
- /app/node_modules/.
- /node_modules/
volumeMounts:
- name: node-modules-volume
mountPath: /node_modules
containers:
- name: frontend
image: YOUR_REGISTRY/YOUR_IMAGE
volumeMounts:
- name: source-volume
mountPath: /app
- name: node-modules-volume
mountPath: /app/node_modules
volumes:
- name: source-volume
hostPath:
path: /YOUR_HOST_CODE_DIRECTORY/frontend
- name: node-modules-volume
emptyDir: {}
Note that the command to copy the node_modules directory is cp -a /SOURCE/. /DEST/ and not the more common cp -r /SOURCE/* /DEST/. The asterisk in the latter command would be interpreted as a literal * instead of a logical * and it wouldn't match every file/directory as intended.
You can do whatever you want with an init container, even create the node_modules directory from a script at initialization (though it'd add a significant delay).
the node_modules is empty
It's empty because you're using emptyDir volume:
An emptyDir volume is first created when a Pod is assigned to a Node,
and exists as long as that Pod is running on that node. As the name
says, it is initially empty.
Application should write data to it.