I want do some device initiation by using Daemonset(K8s resource).
Actually the deivce has been formated(inside container) and mounted(inside container) successfully to a container path /hostmnt/lvpmem/ which is mapped of /mnt/ which is a host path.
mountpoint works fine in container
[root#driver-hm4ll /]
#mountpoint /hostmnt/lvpmem/
/hostmnt/lvpmem/ is a mountpoint
but mountpoint works wrong in host env
[root#host ~]# mountpoint /mnt/lvpmem/
/mnt/lvpmem/ is not a mountpoint
Also the data I write in container under /hostmnt/lvpmem/ can't been seen under /mnt/lvpmem/ in host env.
How can I mount the device so that both host and container can see it ?
Also, if container is destroyed does the mount relation also be destroyed ? I have no idea about umounting the device in host env if mount relation can't be seen.
Some opensource project use nsenter in container to run such format/mount command does it help ?
add /mnt as a volume to pod on directory /hostmnt. So that whatever being written under /hostmnt directory (insisde the container) will be seen on host under directory /mnt .
Example of a pod with hostpath :
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /hostmnt
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /mnt
# this field is optional
type: Directory
Mount point seen in container is just the snapshot when container start. My solution:
Enable hostPID for pod and use nsenter to mount on host
Restart container itself by using exit then the mount point can be seen in new container
Related
Please help! I'm struggling with this for a few days now...
I'm trying to write to a mount in a Kubernetes pod with a non-root user and getting access denied.
In the Kubernetes manifest, I am mounting a windows shared folder like this:
kind: Deployment
apiVersion: apps/v1
metadata:
name: centos-deployment
spec:
template:
spec:
volumes:
- name: windows-mount
flexVolume:
driver: microsoft.com/smb
secretRef:
name: centos-credentials
options:
mountOptions: 'cifs,dir_mode=0777,file_mode=0777'
source: //100.200.300.400/windows-share
containers:
- name: centos-pod
image: 'centos:latest'
command:
- sh
- '-c'
- sleep 1000000
volumeMounts:
- name: windows-mount
mountPath: /var/windows-share
and in the Dockerfile I'm changing to application user like so:
# Drop from 'root' user to 'nobody' (user with no privileges).
USER nobody:nobody
But now, the mount is owned by "root". The "root" user can write to the path but the user "nobody" cannot.
I tried init container to run chmod -R 775 on the folder, but it looks like the root user cannot change the permissions or ownership of the mount. (umask command returned 022)
If I exec into the pod, I can see the mount is set with 755 permissions instead of 777
"file_mode=0755,dir_mode=0755"
[root#centos-deployment-5d46bd8b89-tzghs /]# mount | grep windows-share
//100.200.300.400/windows-share on /var/windows-share type cifs (rw,relatime,vers=default,cache=strict,username=*******,domain=,uid=0,noforceuid,gid=0,noforcegid,addr=100.200.300.400,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
Any idea how to mount a Windows share so that it is writable by non-root user?
Thanks! any help will be very appreciated
Full reference: https://linux.die.net/man/8/mount.cifs
Try playing with the mountOptions. For example:
uid=arg - sets the uid that will own all files or directories on the mounted filesystem when the server does not provide ownership information. It may be specified as either a username or a numeric uid. When not specified, the default is uid 0. The mount.cifs helper must be at version 1.10 or higher to support specifying the uid in non-numeric form. See the section on FILE AND DIRECTORY OWNERSHIP AND PERMISSIONS below for more information.
volumes:
- name: windows-mount
...
options:
mountOptions: 'cifs,uid=<YOUR_USERNAME>,dir_mode=0777,file_mode=0777'
If this doesen't work you can also try adding noperm mount option.
noperm - Client does not do permission checks. This can expose files on this mount to access by other users on the local client system. It is typically only needed when the server supports the CIFS Unix Extensions but the UIDs/GIDs on the client and server system do not match closely enough to allow access by the user doing the mount. Note that this does not affect the normal ACL check on the target machine done by the server software (of the server ACL against the user name provided at mount time).
volumes:
- name: windows-mount
...
options:
mountOptions: 'cifs,dir_mode=0777,file_mode=0777,noperm'
About using chmod/chown on CIFS
The core CIFS protocol does not provide unix ownership information or mode for files and directories. Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. Attempting to change these values via chmod/chown will return success but have no effect.
I'm trying to run Nginx as the docker container on SELinux enabled server with mounted configuration as
-v /host/path/nginx.conf:/etc/nginx/nginx.conf:Z
That works fine till I will deploy updated configuration using ansible playbook (user/group root and mode 0644) as:
tasks:
- name: remove NGINX conf
file:
path: "/host/path/nginx.conf"
state: absent
become: true
- name: copy NGINX conf
become: true
copy:
src: "../nginx-conf/nginx.conf"
dest: "/host/path/nginx.conf"
mode: 0640
owner: root
group: root
and then reload nginx in the container as
docker container exec nginx-service nginx -s reload
that will error out with Permission Denied (13) to deployed configuration.
When I will check permissions to /host/path/nginx.conf those are correct (user/group:root and mode 0644) but for some reason nginx is not able to read that file.
When I will get inside the container and check /etc/nginx/nginx.conf permissions there are not carried over - they show up as question marks.
Any clue how to get it to work without reloading whole container just nginx service?
I am trying to mount a hostPath volume into a Kubernetes Pod. An example of a hostPath volume specification is shown below, which is taken from the docs. I am deploying to hosts that are running RHEL 7 with SELinux enabled.
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
When my Pod tries to read from a file that has been mounted from the underlying host, I get a "Permission Denied" error. When I run setenforce 0 to turn off SELinux, the error goes away and I can access the file. I get the same error when I bind mount a directory into a Docker container.
The issue is described here and, when using Docker, can be fixed by using the z or Z bind mount flag, described in the Docker docs here.
Whilst I can fix the issue by running
chcon -Rt svirt_sandbox_file_t /path/to/my/host/dir/to/mount
I see this as a nasty hack, as I need to do this on every host in my Kubernetes cluster and also because my deployment of Kubernetes as described in the YAML spec is not a complete description of what it is that needs to be done to get my YAML to run correctly. Turning off SELinux is not an option.
I can see that Kubernetes mentions SELinux security contexts in the docs here, but I haven't been able to successfully mount a hostPath volume into a pod without getting the permission denied error.
What does the YAML need to look like to successfully enable a container to mount a HostPath volume from an underlying host that is running SELinux?
Update:
The file I am accessing is a CA certificate that has these labels:
system_u:object_r:cert_t:s0
When I use the following options:
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
and then check the access control audit errors via ausearch -m avc -ts recent, I can see that there is a permission denied error where the container has a level label of s0:c123,c456, so I can see that the level label works. I have set the label to be s0.
However, if I try to change the type label to be cert_t, the container doesn't even start, there's an error :
container_linux.go:247: starting container process caused "process_linux.go:364: container init caused \"write /proc/self/task/1/attr/exec: invalid argument\""
I don't seem to be able to change the type label of the container.
Expanding on the answer from VAS as it is partially correct:
You can only specify the level portion of an SELinux label when relabeling a path destination pointed to by a hostPath volume. This is automatically done so by the seLinuxOptions.level attribute specified in your securityContext.
However attributes such as seLinuxOptions.type currently have no effect on volume relabeling. As of this writing, this is still an open issue within Kubernetes
You can assign SELinux labels using seLinuxOptions:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
securityContext:
seLinuxOptions: # it may don’t have the desired effect
level: "s0:c123,c456"
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
According to documentation:
Thanks to Phil for pointing that out. It appears to be working only in Pod.spec.securityContext according to the issue comment
seLinuxOptions: Volumes that support SELinux labeling are relabeled to be accessible by the label specified under seLinuxOptions. Usually you only need to set the level section. This sets the Multi-Category Security (MCS) label given to all Containers in the Pod as well as the Volumes.
You could try with full permissions:
...
image: k8s.gcr.io/test-webserver
securityContext:
privileged: true
...
Using selinux can solve this problem. Reference article:
https://zhimin-wen.medium.com/selinux-policy-for-openshift-containers-40baa1c86aa5
In addition: You can refer to the selinux parameters to set the addition, deletion, and modification of the mount directory
https://selinuxproject.org/page /ObjectClassesPerms
my setting：
If a directory's selinux is unconfined_u:object_r:kubernetes_file_t:s0 , you kan defind a selinux policy is:
module myapp 1.0;
require {
type kubernetes_file_t;
type container_t;
class file { create open read unlink write getattr execute setattr link };
class dir { add_name create read remove_name write };
}
#============= container_t ==============
#!!!! This avc is allowed in the current policy
allow container_t kubernetes_file_t:dir { add_name create read remove_name write };
#!!!! This avc is allowed in the current policy
allow container_t kubernetes_file_t:file { create open read unlink write getattr execute setattr link };
run command on node:
sudo checkmodule -M -m -o myapp.mod myapp.te
sudo semodule_package -o myapp.pp -m myapp.mod
sudo semodule -i myapp.pp
Objective
Know how to trouble shoot and what knowledge is required to trouble shoot permission issues of Docker container accessing host files.
Problem
Access to /var/run/docker.sock mounted inside a OpenShift container via hostPath causes permission denied. The issue does not happen if the same container is deployed to K8S 1.9.x, hence it is OpenShift specific issue.
[ec2-user#ip-10-0-4-62 ~]$ ls -laZ /var/run/docker.sock
srw-rw----. root docker system_u:object_r:container_var_run_t:s0 /var/run/docker.sock
[ec2-user#ip-10-0-4-62 ~]$ docker exec 9d0c6763d855 ls -laZ /var/run/docker.sock
srw-rw----. 1 root 1002 system_u:object_r:container_var_run_t:s0 0 Jan 16 09:54 /var/run/docker.sock
https://bugzilla.redhat.com/show_bug.cgi?id=1244634 says svirt_sandbox_file_t SELinux label is required for RHEL, so changed the label.
$ chcon -Rt container_runtime_t docker.sock
[ec2-user#ip-10-0-4-62 ~]$ ls -aZ /var/run/docker.sock
srw-rw----. root docker system_u:object_r:svirt_sandbox_file_t:s0 /var/run/docker.sock
Redeploy the container but still permission denied.
$ docker exec -it 9d0c6763d855 curl -ivs --unix-socket /var/run/docker.sock http://localhost/version
* Trying /var/run/docker.sock...
* Immediate connect fail for /var/run/docker.sock: Permission denied
* Closing connection 0
OpenShift by default does not allow hostPath so it was addressed.
oc adm policy add-scc-to-user privileged system:serviceaccount:{{ DATADOG_NAMESPACE }}:{{ DATADOG_SERVICE_ACCOUNT }}
I suppose SELinux or OpenShift SCC or other container/docker permission is causing this but need a clue how to find the cause.
Openshift requires special permissions for in order to allow pods to use volumes in nodes.
Do the following:
Create standard security-context yaml:
kind: SecurityContextConstraints
apiVersion: v1
metadata:
name: scc-hostpath
allowPrivilegedContainer: true
runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
fsGroup:
type: RunAsAny
supplementalGroups:
type: RunAsAny
users:
- my-admin-user
groups:
- my-admin-group
oc create -f scc-hostpath.yam
Add the "allowHostDirVolumePlugin" privilege to this security-context:
oc patch scc scc-hostpath -p '{"allowHostDirVolumePlugin": true}'
Associate the pod's service account with the above security context
oc adm policy add-scc-to-user scc-hostpath system:serviceaccount:<service_account_name>
I am using kubernetes on a single machine for testing, I have built a custom image from the nginx docker image, but when I try to use the image in kubernetes I get an image pull error?????
MY POD YAML
kind: Pod
apiVersion: v1
metadata:
name: yumserver
labels:
name: frontendhttp
spec:
containers:
- name: myfrontend
image: my/nginx:latest
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: mypd
imagePullSecrets:
- name: myregistrykey
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim-1
MY KUBERNETES COMMAND
kubectl create -f pod-yumserver.yaml
THE ERROR
kubectl describe pod yumserver
Name: yumserver
Namespace: default
Image(s): my/nginx:latest
Node: 127.0.0.1/127.0.0.1
Start Time: Tue, 26 Apr 2016 16:31:42 +0100
Labels: name=frontendhttp
Status: Pending
Reason:
Message:
IP: 172.17.0.2
Controllers: <none>
Containers:
myfrontend:
Container ID:
Image: my/nginx:latest
Image ID:
QoS Tier:
memory: BestEffort
cpu: BestEffort
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
mypd:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: myclaim-1
ReadOnly: false
default-token-64w08:
Type: Secret (a secret that should populate this volume)
SecretName: default-token-64w08
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
13s 13s 1 {default-scheduler } Normal Scheduled Successfully assigned yumserver to 127.0.0.1
13s 13s 1 {kubelet 127.0.0.1} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
12s 12s 1 {kubelet 127.0.0.1} spec.containers{myfrontend} Normal Pulling pulling image "my/nginx:latest"
8s 8s 1 {kubelet 127.0.0.1} spec.containers{myfrontend} Warning Failed Failed to pull image "my/nginx:latest": Error: image my/nginx:latest not found
8s 8s 1 {kubelet 127.0.0.1} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "myfrontend" with ErrImagePull: "Error: image my/nginx:latest not found"
So you have the image on your machine aready. It still tries to pull the image from Docker Hub, however, which is likely not what you want on your single-machine setup. This is happening because the latest tag sets the imagePullPolicy to Always implicitly. You can try setting it to IfNotPresent explicitly or change to a tag other than latest. – Timo Reimann Apr 28 at 7:16
For some reason Timo Reimann did only post this above as a comment, but it definitely should be the official answer to this question, so I'm posting it again.
Run eval $(minikube docker-env) before building your image.
Full answer here: https://stackoverflow.com/a/40150867
This should work irrespective of whether you are using minikube or not :
Start a local registry container:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Do docker images to find out the REPOSITORY and TAG of your local image. Then create a new tag for your local image :
docker tag <local-image-repository>:<local-image-tag> localhost:5000/<local-image-name>
If TAG for your local image is <none>, you can simply do:
docker tag <local-image-repository> localhost:5000/<local-image-name>
Push to local registry :
docker push localhost:5000/<local-image-name>
This will automatically add the latest tag to localhost:5000/<local-image-name>.
You can check again by doing docker images.
In your yaml file, set imagePullPolicy to IfNotPresent :
...
spec:
containers:
- name: <name>
image: localhost:5000/<local-image-name>
imagePullPolicy: IfNotPresent
...
That's it. Now your ImagePullError should be resolved.
Note: If you have multiple hosts in the cluster, and you want to use a specific one to host the registry, just replace localhost in all the above steps with the hostname of the host where the registry container is hosted. In that case, you may need to allow HTTP (non-HTTPS) connections to the registry:
5 (optional). Allow connection to insecure registry in worker nodes:
sudo echo '{"insecure-registries":["<registry-hostname>:5000"]}' > /etc/docker/daemon.json
just add imagePullPolicy to your deployment file
it worked for me
spec:
containers:
- name: <name>
image: <local-image-name>
imagePullPolicy: Never
The easiest way to further analysis ErrImagePull problems is to ssh into the node and try to pull the image manually by doing docker pull my/nginx:latest. I've never set up Kubernetes on a single machine but could imagine that the Docker daemon isn't reachable from the node for some reason. A handish pull attempt should provide more information.
If you are using a vm driver, you will need to tell Kubernetes to use the Docker daemon running inside of the single node cluster instead of the host.
Run the following command:
eval $(minikube docker-env)
Note - This command will need to be repeated anytime you close and restart the terminal session.
Afterward, you can build your image:
docker build -t USERNAME/REPO .
Update, your pod manifest as shown above and then run:
kubectl apply -f myfile.yaml
in your case your yaml file should have
imagePullPolicy: Never
see below
kind: Pod
apiVersion: v1
metadata:
name: yumserver
labels:
name: frontendhttp
spec:
containers:
- name: myfrontend
image: my/nginx:latest
imagePullPolicy: Never
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: mypd
imagePullSecrets:
- name: myregistrykey
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim-1
found this here
https://keepforyourself.com/docker/run-a-kubernetes-pod-locally/
Are you using minikube on linux? You need to install docker ( I think), but you don't need to start it. Minikube will do that. Try using the KVM driver with this command:
minikube start --vm-driver kvm
Then run the eval $(minikube docker-env) command to make sure you use the minikube docker environment. build your container with a tag build -t mycontainername:version .
if you then type docker ps you should see a bunch of minikube containers already running.
kvm utils are probably already on your machine, but they can be installed like this on centos/rhel:
yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python
Make sure that your "Kubernetes Context" in Docker Desktop is actually a "docker-desktop" (i.e. not a remote cluster).
(Right click on Docker icon, then select "Kubernetes" in menu)
All you need to do is just do a docker build from your dockerfile, or get all the images on the nodes of your cluster, do a suitable docker tag and create the manifest.
Kubernetes doesn't directly pull from the registry. First it searches for the image on local storage and then docker registry.
Pull latest nginx image
docker pull nginx
docker tag nginx:latest test:test8970
Create a deployment
kubectl run test --image=test:test8970
It won't go to docker registry to pull the image. It will bring up the pod instantly.
And if image is not present on local machine it will try to pull from docker registry and fail with ErrImagePull error.
Also if you change the imagePullPolicy: Never. It will never look for the registry to pull the image and will fail if image is not found with error ErrImageNeverPull.
kind: Deployment
metadata:
labels:
run: test
name: test
spec:
replicas: 1
selector:
matchLabels:
run: test
template:
metadata:
creationTimestamp: null
labels:
run: test
spec:
containers:
- image: test:test8070
name: test
imagePullPolicy: Never
Adding another answer here as the above gave me enough to figure out the cause of my particular instance of this issue. Turns out that my build process was missing the tagging needed to make :latest work. As soon as I added a <tags> section to my docker-maven-plugin configuration in my pom.xml, everything was hunky-dory. Here's some example configuration:
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.27.2</version>
<configuration>
<images>
<image>
<name>akka-cluster-demo:${docker.image.version}</name>
<build>
<from>openjdk:8-jre-alpine</from>
Adding this:
<tags>
<tag>latest</tag>
<tag>${git.commit.version}</tag>
</tags>
The rest continues as before:
<ports>
<port>8080</port>
<port>8558</port>
<port>2552</port>
</ports>
<entryPoint>
<exec>
<args>/bin/sh</args>
<args>-c</args>
<args>java -jar /maven/cluster-sharding-kubernetes.jar</args>
</exec>
</entryPoint>
<assembly>
<inline>
<dependencySets>
<dependencySet>
<useProjectAttachments>true</useProjectAttachments>
<includes>
<include>akka-java:cluster-sharding-kubernetes:jar:allinone</include>
</includes>
<outputFileNameMapping>cluster-sharding-kubernetes.jar</outputFileNameMapping>
</dependencySet>
</dependencySets>
</inline>
</assembly>
</build>
</image>
</images>
</configuration>
</plugin>
ContainerD (and Windows)
I had the same error, while trying to run a custom windows container on a node. I had imagePullPolicy set to Never and a locally existing image present on the node. The image also wasn't tagged with latest, so the comment from Timo Reimann wasn't relevant.
Also, on the node machine, the image showed up when using nerdctl image. However they didn't show up in crictl images.
Thanks to a comment on Github, I found out that the actual problem is a different namespace of ContainerD.
As shown by the following two commands, images are not automatically build in the correct namespace:
ctr -n default images ls # shows the application images (wrong namespace)
ctr -n k8s.io images ls # shows the base images
To solve the problem, export and reimport the images to the correct namespace k8s.io by using the following command:
ctr --namespace k8s.io image import exported-app-image.tar
I was facing similar issue .Image was present in local but k8s was not able to pick it up.
So I went to terminal ,deleted the old image and ran eval $(minikube -p minikube docker-env) command.
Rebuilt the image and the redeployed the deployment yaml ,and it worked