I'm using Kubernetes 1.15.7 and my issue is similar to https://github.com/kubernetes/kubernetes/issues/62362
apiVersion: v1
kind: Pod
metadata:
name: init-demo
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
# These containers are run during pod initialization
restartPolicy: Always
initContainers:
- name: install
image: busybox
command:
- sh
- -c
- sleep 60
volumeMounts:
- name: workdir
mountPath: "/work-dir"
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}
On the node the container is runner if I issue a docker container prune ,it removes the exited busybox (init) container. Only to restart it again and trigger the pod to restart too.
I found the github issue similar to this but without much explaination. These exited container as such do not show up to consumer much same using docker system df but it doesnt allow me to run the prune command as a whole on the node.
Kubelet manages garbage collection of docker images so you dont have to.
Take a look at k8s documentation for more info on this topic.
From k8s documentation:
Garbage collection is a helpful function of kubelet that will clean up unused images and unused containers. Kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.
External garbage collection tools are not recommended as these tools can potentially break the behavior of kubelet by removing containers expected to exist
Related
I have a pod running Linux, I have let others use it. Now I need to save the changes made by others. Since sometimes I need to delete/restart the pod, the changes are reverted and new pod get created. So I want to save the pod container as docker image and use that image to create a pod.
I have tried kubectl debug node/pool-89899hhdyhd-bygy -it --image=ubuntu then install docker, dockerd inside but they don't have root permission to perform operations, installed crictl they where listing the containers but they don't have options to save them.
Also created a privileged docker image, created a pod from it, then used the command kubectl exec --stdin --tty app-7ff786bc77-d5dhg -- /bin/sh then tried to get running container, but it was not listing the containers. Below is the deployment i used to the privileged docker container
kind: Deployment
apiVersion: apps/v1
metadata:
name: app
labels:
app: backend-app
backend-app: app
spec:
replicas: 1
selector:
matchLabels:
app: backend-app
task: app
template:
metadata:
labels:
app: backend-app
task: app
spec:
nodeSelector:
kubernetes.io/hostname: pool-58i9au7bq-mgs6d
volumes:
- name: task-pv-storage
hostPath:
path: /run/docker.sock
type: Socket
containers:
- name: app
image: registry.digitalocean.com/my_registry/docker_app#sha256:b95016bd9653631277455466b2f60f5dc027f0963633881b5d9b9e2304c57098
ports:
- containerPort: 80
volumeMounts:
- name: task-pv-storage
mountPath: /var/run/docker.sock
Is there any way I can achieve this, get the pod container and save it as a docker image? I am using digitalocean to run my kubernetes apps, I do not ssh access to the node.
This is not a feature of Kubernetes or CRI. Docker does support snapshotting a running container to an image however Kubernetes no longer supports Docker.
Thank you all for your help and suggestions. I found a way to achieve it using the tool nerdctl - https://github.com/containerd/nerdctl.
I have a docker image that creates few folders and extract files into it like below
RUN mkdir -p /home/myapp/myappv4 \
/home/myapp/myappv4/files \
/home/myapp/myappv4/files/logs \
/home/myapp/myappv4/myappentries
WORKDIR /home/myapp
RUN chown -R myapp:myapp /home/myapp
ADD /myapp-v4-files/*.zip /home/myapp/myappv4/files/
ADD /myapp-v4-files/init.txt /home/myapp/myappv4/myappentries/
ADD /myapp-v4-files/pro.json /home/myapp/myappv4/myappentries/
These folders and files needs to be accessed by other containers in a pod in kubernetes. Should i create persistentvolume in kubernetes and have these locations in them and copy the content from this container to this volume? In that way they would not get deleted right?. Since i am new to kubernetes i am not sure on how to achieve this. Transition from docker container to kubernetes deployment seems to be a confusing part for me,any help on this would be appreciated.
If you want to share a set of directories between multiple containers in a single pod, using only EmptyDir volume will suffice. You don't need to use PersistentVolumes (unless you want persistence, meaning you want the data to survive pod restarts).
However note that adding a volume (a kubernetes construct) will overwrite the files already present in your container at the path where you are mounting the volume, kind of what happens with a layered filesystem that docker uses.
For your usecase, I think you can move the file fetching logic from the Dockerfile to a script that the pod will run, that will fix the above mentioned issue.
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /cache
name: cache-volume
- image: k8s.gcr.io/test-webserver
name: test-container-2
volumeMounts:
- mountPath: /cache
name: cache-volume
volumes:
- name: cache-volume
emptyDir: {}
Read more about volumes here.
I'm using the kubernetes plugin to setup a pipeline on jenkins to compile some code.
MY GOAL:
In this pipeline, I'm trying to access some data from a docker container to use it as a cache in a second on (as shown below).
apiVersion: v1
kind: Pod
metadata:
name: cache-test
spec:
restartPolicy: Never
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: cache-container
image: cache:latest
volumeMounts:
- name: shared-data
mountPath: /cache
command:
- cat
- name: debian-container
image: debian
volumeMounts:
- name: shared-data
mountPath: /pod-data
command:
- cat
PROBLEM:
My issue is that when I mount my shared-folder in /cache directly, all my data get erased (overwritten).
WORK AROUND:
One work around would be to to create an intermediate directory where I can copy my data:
apiVersion: v1
kind: Pod
metadata:
name: cache-test
spec:
restartPolicy: Never
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: cache-container
image: cache:latest
volumeMounts:
- name: shared-data
mountPath: /shared-folder
command:
- cat
- name: debian-container
image: debian
volumeMounts:
- name: shared-data
mountPath: /pod-data
command:
- cat
And the in my Jenkins pipeline add this step:
container('cache-container') {
sh """#!/usr/bin/env bash
set -exu
cp -r /cache/* /shared-folder
"""
} // container
QUESTION:
Is there a way to avoid this copy step? Maybe a kubernetes volume setup that doesn't overwrite what's in the container?
I went through the documentation couple times without finding anything..
You can't use a container as a cache without copying data to a volume shared with other containers. But you probably shouldn't anyway.
You probably want to move your cache out of the container and make it a PersistentVolume. PersistentVolumeClaims can claim PV and then pods can mount these PVCs.
The problem with PVs, no matter static or dynamic they are - they won't be Available for PVCs when previously locking PVC was deleted. They will stuck in Released state. Kubernetes doesn't do it automatically for a reason - workloads aren't supposed to have access to data from other workloads. For when they do - the idiomatic Kubernetes way to do it is StatefulSets, so Kubernetes guarantees that only the replicas of the same workload may claim the old data. Unfortunately it's just doesn't work for a build caches.
I wrote two simple controllers - automatic PV releaser (that would find and make Released PVs Available again for new PVCs) and dynamic PVC provisioner (for Jenkins Kubernetes plugin specifically - so you can define a PVC as annotation on a Pod). Check it out here https://github.com/plumber-cd/kubernetes-dynamic-reclaimable-pvc-controllers. There is a full Jenkinsfile example here https://github.com/plumber-cd/kubernetes-dynamic-reclaimable-pvc-controllers/tree/main/examples/jenkins-kubernetes-plugin-with-build-cache.
From the docker hub there is an image which is maintained by amazon.
Any one know how to configure and start the container as I cannot find any documentation
I got this working! I was having the same issue with you when you see Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ... Cannot access /etc/cwagentconfig: lstat /etc/cwagentconfig: no such file or directoryValid Json input schema.
What you need to do is put your config file in /etc/cwagentconfig. A functioning dockerfile:
FROM amazon/cloudwatch-agent:1.230621.0
COPY config.json /etc/cwagentconfig
Where config.json is some cloudwatch agent configuration, such as given by LinPy's answer.
You can ignore the warning about /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json, or you can also COPY the config.json file to that location in the dockerfile as well.
I will also share how I found this answer:
I needed this run in ECS as a sidecar, and I could only find docs on how to run it in kubernetes. Following this documentation: https://docs.aws.amazon.com/en_pv/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-StatsD.html I decided to download all the example k8s manifests, when I saw this one:
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: amazonlinux
spec:
containers:
- name: amazonlinux
image: amazonlinux
command: ["/bin/sh"]
args: ["-c", "sleep 300"]
- name: cloudwatch-agent
image: amazon/cloudwatch-agent
imagePullPolicy: Always
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 200m
memory: 100Mi
volumeMounts:
- name: cwagentconfig
mountPath: /etc/cwagentconfig
volumes:
- name: cwagentconfig
configMap:
name: cwagentstatsdconfig
terminationGracePeriodSeconds: 60
So I saw that the volume mount cwagentconfig mounts to /etc/cwagentconfig and that's from the cwagentstatsdconfig configmap, and that's just the json file.
You just to run the container with log-opt, as the log agent is the main process of the container.
docker run --log-driver=awslogs --log-opt awslogs-region=us-west-2 --log-opt awslogs-group=myLogGroup amazon/cloudwatch-agent
You can find more details here and here.
I do not know why you need an agent in a container, but the best practice is to send each container log directly to cloud watch using aws log driver.
Btw this is entrypoint of the container.
"Entrypoint": [
"/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent"
],
All you need to call
/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent
Here is how I got it working in our Docker containers without systemctl or System V init.
This is from official Documentation:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:configuration-file-path -s
here the Docs:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html#start-CloudWatch-Agent-EC2-commands-fleet
Installation path may be different, but that is how the agent is started as per docs.
I'm looking for a pattern that allows to share volumes between two containers running on the same pod in Kubernetes.
My use case is:
I have a Ruby on Rails application running inside a docker container.
The docker image contains static assets in /app/<app-name>/public directory, and I need to access those assets from the nginx container running alongside in the same pod.
In 'vanilla' docker I would have used --volumes-from flag to share this directory:
docker run --name app -v /app/<app-dir>/public <app-image>
docker run --volumes-from app nginx
After reading this doc: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/volumes.md
I tried this (only relevant entries presented):
spec:
containers:
- image: <app-image>
name: <app-name>
volumeMounts:
- mountPath: /app/<app-name>/public
name: assets
- image: nginx
name: nginx
volumeMounts:
- mountPath: /var/www/html
name: assets
readOnly: true
volumes:
- name: assets
hostPath:
path: /tmp/assets
But:
Even though /tmp/assets on the node exists, it's empty
/app/<app-name>/public inside the app container is also empty
As a workaround I'm gonna try to populate the shared directory when the application container is up (simply cp /app/<app-name>/public/* to shared directory), but I really dislike this idea.
Question: how to mimic --volumes-from in Kubernetes, or if there is no direct counterpart, how can I share files from one container to other running in the same pod ?
apiVersion: v1beta3
Client Version: version.Info{Major:"0", Minor:"17", GitVersion:"v0.17.0", GitCommit:"82f8bdac06ddfacf493a9ed0fedc85f5ea62ebd5", GitTreeState:"clean"}
Server Version: version.Info{Major:"0", Minor:"17", GitVersion:"v0.17.0", GitCommit:"82f8bdac06ddfacf493a9ed0fedc85f5ea62ebd5", GitTreeState:"clean"}
[update-2016-8] In latest Kubernetes release, you can use a very nice feature named init-container to replace the postStart part in my answer below, which will make sure the container order.
apiVersion: v1
kind: Pod
metadata:
name: javaweb-2
spec:
initContainers:
- name: war
image: resouer/sample:v2
command: ["cp", "/sample.war", "/app"]
volumeMounts:
- mountPath: /app
name: app-volume
containers:
- name: tomcat
image: resouer/mytomcat:7.0
command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
volumeMounts:
- mountPath: /root/apache-tomcat-7.0.42-v2/webapps
name: app-volume
ports:
- containerPort: 8080
hostPort: 8001
volumes:
- name: app-volume
emptyDir: {}
NOTE: initContainer is still a beta feature so the work version of this yaml is actually like: http://kubernetes.io/docs/user-guide/production-pods/#handling-initialization, please notice the pod.beta.kubernetes.io/init-containers part.
---original answer begin---
Actually, you can. You need to use container life cycle handler to control what files/dirs you want to share with other containers. Like:
---
apiVersion: v1
kind: Pod
metadata:
name: server
spec:
restartPolicy: OnFailure
containers:
- image: resouer/sample:v2
name: war
lifecycle:
postStart:
exec:
command:
- "cp"
- "/sample.war"
- "/app"
volumeMounts:
- mountPath: /app
name: hostv1
- name: peer
image: busybox
command: ["tail", "-f", "/dev/null"]
volumeMounts:
- name: hostv2
mountPath: /app/sample.war
volumes:
- name: hostv1
hostPath:
path: /tmp
- name: hostv2
hostPath:
path: /tmp/sample.war
Please check my gist for more details:
https://gist.github.com/resouer/378bcdaef1d9601ed6aa
And of course you can use emptyDir. Thus, war container can share its /sample.war to peer container without mess peer's /app directory.
If we can tolerate /app been overridden, it will be much simpler:
---
apiVersion: v1
kind: Pod
metadata:
name: javaweb-2
spec:
restartPolicy: OnFailure
containers:
- image: resouer/sample:v2
name: war
lifecycle:
postStart:
exec:
command:
- "cp"
- "/sample.war"
- "/app"
volumeMounts:
- mountPath: /app
name: app-volume
- image: resouer/mytomcat:7.0
name: tomcat
command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
volumeMounts:
- mountPath: /root/apache-tomcat-7.0.42-v2/webapps
name: app-volume
ports:
- containerPort: 8080
hostPort: 8001
volumes:
- name: app-volume
emptyDir: {}
The answer is - for now - you can't. Here's a couple of discussion threads from the Kubernetes issues:
https://github.com/GoogleCloudPlatform/kubernetes/issues/6120
https://github.com/GoogleCloudPlatform/kubernetes/issues/831
However, may I suggest that you have an alternate design that might work better?
If your assets are locked at the point of the container going live,
you could use something like gitRepo
volume which would copy it to an emptyDir at the point of going live, and would mean you wouldn't have to move the content around at
all, just download it directly to the shared directory.
If your assets are locked at the point of the container
being built, it's probably best to copy them in at that point, using
the Docker COPY command.
If you really want to stick with the way you're doing it, you would have to copy the content to the emptyDir volume, which is designed for exactly what you're looking for (minus the lack of having to copy it in).
NFS[1] volumes also could solve your problem, but may be overly complex.
Additionally, I'd recommend that these two services exist in different pods, so you can scale each separately. You can create a service endpoint to communicate between them if you need to.
[1] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/nfs/nfs-web-pod.yaml
Further update from the future:
There is now a FlexVol plugin for Docker volumes: https://github.com/dims/docker-flexvol
At the time of writing, FlexVol is still an alpha feature though, so caveat emptor.
Kubernetes has its own volume types and these are most used volume type:
emptyDir
secret
gitRepo
hostPath (similar to --volumes-from)
config Maps
persistent storage (storage disks provided by cloud platforms)
You can find more about kubernets volumes here -https://kubernetes.io/docs/concepts/storage/volumes/
an example of hostpath volume :
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
hostpath will mount host/node directory to container directory.Multiple containers inside a pod can use different or same volumes.You need to mention it in each container.
hostPath volumes are independent of pod lifecycle but it create tight coupling between node and pod , you should avoid using hostPath.
If you are using Docker v17.0.5 or greater you can use a multi-stage build to copy files from one of your containers to the other during build time. This is a great primer on the advanced features at https://medium.com/#tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae
The way I used it to copy static assets from my backend container into Nginx proxy is
ARG API_BACKEND_CONTAINER="api:backend"
FROM $API_BACKEND_CONTAINER as source
FROM nginx:mainline-alpine
ARG NGINX_ROOT=/usr/share/nginx/html/
COPY --from=source /var/share/api/static/ ${NGINX_ROOT}
The great thing is that because the API_BACKEND_CONTAINER is a build arg I'm able to pass in the tag of the latest API build.