container labels in kubernetes - docker

I am building my docker image with jenkins using:
docker build --build-arg VCS_REF=$GIT_COMMIT \
--build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
--build-arg BUILD_NUMBER=$BUILD_NUMBER -t $IMAGE_NAME\
I was using Docker but I am migrating to k8.
With docker I could access those labels via:
docker inspect --format "{{ index .Config.Labels \"$label\"}}" $container
How can I access those labels with Kubernetes ?
I am aware about adding those labels in .Metadata.labels of my yaml files but I don't like it that much because
- it links those information to the deployment and not the container itself
- can be modified anytime
...
kubectl describe pods
Thank you

Kubernetes doesn't expose that data. If it did, it would be part of the PodStatus API object (and its embedded ContainerStatus), which is one part of the Pod data that would get dumped out by kubectl get pod deployment-name-12345-abcde -o yaml.
You might consider encoding some of that data in the Docker image tag; for instance, if the CI system is building a tagged commit then use the source control tag name as the image tag, otherwise use a commit hash or sequence number. Another typical path is to use a deployment manager like Helm as the principal source of truth about deployments, and if you do that there can be a path from your CD system to Helm to Kubernetes that can pass along labels or annotations. You can also often set up software to know its own build date and source control commit ID at build time, and then expose that information via an informational-only API (like an HTTP GET /_version call or some such).

I'll add another option.
I would suggest reading about the Recommended Labels by K8S:
Key Description
app.kubernetes.io/name The name of the application
app.kubernetes.io/instance A unique name identifying the instance of an application
app.kubernetes.io/version The current version of the application (e.g., a semantic version, revision hash, etc.)
app.kubernetes.io/component The component within the architecture
app.kubernetes.io/part-of The name of a higher level application this one is part of
app.kubernetes.io/managed-by The tool being used to manage the operation of an application
So you can use the labels to describe a pod:
apiVersion: apps/v1
kind: Pod # Or via Deployment
metadata:
labels:
app.kubernetes.io/name: wordpress
app.kubernetes.io/instance: wordpress-abcxzy
app.kubernetes.io/version: "4.9.4"
app.kubernetes.io/managed-by: helm
app.kubernetes.io/component: server
app.kubernetes.io/part-of: wordpress
And use the downward api (which works in a similar way to reflection in programming languages).
There are two ways to expose Pod and Container fields to a running Container:
1 ) Environment variables.
2 ) Volume Files.
Below is an example for using volumes files:
apiVersion: v1
kind: Pod
metadata:
name: kubernetes-downwardapi-volume-example
labels:
version: 4.5.6
component: database
part-of: etl-engine
annotations:
build: two
builder: john-doe
spec:
containers:
- name: client-container
image: k8s.gcr.io/busybox
command: ["sh", "-c"]
args: # < ------ We're using the mounted volumes inside the container
- while true; do
if [[ -e /etc/podinfo/labels ]]; then
echo -en '\n\n'; cat /etc/podinfo/labels; fi;
if [[ -e /etc/podinfo/annotations ]]; then
echo -en '\n\n'; cat /etc/podinfo/annotations; fi;
sleep 5;
done;
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
volumes: # < -------- We're mounting in our example the pod's labels and annotations
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations
Notice that in the example we accessed the labels and annotations that were passed and mounted to the /etc/podinfo path.
Beside labels and annotations, the downward API exposed multiple additional options like:
The pod's IP address.
The pod's service account name.
The node's name and IP.
A Container's CPU limit , CPU request , memory limit, memory request.
See full list in here.
(*) A nice blog discussing the downward API.
(**) You can view all your pods labels with
$ kubectl get pods --show-labels
NAME ... LABELS
my-app-xxx-aaa pod-template-hash=...,run=my-app
my-app-xxx-bbb pod-template-hash=...,run=my-app
my-app-xxx-ccc pod-template-hash=...,run=my-app
fluentd-8ft5r app=fluentd,controller-revision-hash=...,pod-template-generation=2
fluentd-fl459 app=fluentd,controller-revision-hash=...,pod-template-generation=2
kibana-xyz-adty4f app=kibana,pod-template-hash=...
recurrent-tasks-executor-xaybyzr-13456 pod-template-hash=...,run=recurrent-tasks-executor
serviceproxy-1356yh6-2mkrw app=serviceproxy,pod-template-hash=...
Or viewing only specific label with $ kubectl get pods -L <label_name>.

Related

How to expose low-numbered ports in the kubernetes mini-cluster that comes with Docker Desktop

I'm using the kubernetes cluster built in to Docker Desktop to develop my application.
I would like to expose services inside the cluster as ports on localhost.
I can do so using kubectl expose deployment foobar --type=NodePort --port=30088, which creates a service like this:
apiVersion: v1
kind: Service
metadata:
labels:
role: web
name: foobar
spec:
externalTrafficPolicy: Cluster
ports:
- nodePort: 30088
port: 80
protocol: TCP
targetPort: 80
selector:
role: web
type: NodePort
But it only works for very high numbered ports. If I try something lower I get:
The Service "kafka-external" is invalid: spec.ports[0].nodePort: Invalid value: 9092: provided port is not in the valid range. The range of valid ports is 30000-32767
It seems there is a kubernetes apiserver setting called ServiceNodePortRange which would allow me to override this restriction, but I can't figure out how to set it on Docker's builtin cluster.
So my question is: how do I expose a specific, low-numbered port (like 9092) on Docker's kubernetes cluster? Is there a way to override that setting? Or a better way to expose the service than NodePort?
NodePort is intended to be a building block for load-balancers or other
ingress modes. This means it didn't matter which port you got as long as
you got one. This makes it a little clunky to use directly - you can't
have just any port. You can change the port range, but you run the risk of
conflicts with real things running on your nodes and with any pod HostPorts.
The default range is indeed 30000-32767 but it can be changed by setting the --service-node-port-range Update the file /etc/kubernetes/manifests/kube-apiserver.yaml and add the line --service-node-port-range=xxxxx-yyyyy.
In the Kubernetes cluster there is a kube-apiserver.yaml file which is in the directory - /etc/kubernetes/manifests/kube-apiserver.yaml but not on the kube-apiserver container/pod but on the master itself.
Login to Docker VM:
Add the following line to the pod spec:
spec:
containers:
- command:
- kube-apiserver
...
- --service-node-port-range=xxxxx-yyyyy # <-- add this line
...
Save and exit. Pod kube-apiserver will be restarted with new parameters.
Exit Docker VM (for screen: Ctrl-a,k , for container: Ctrl-d )
Check the results:
$ kubectl get pod kube-apiserver-docker-desktop -o yaml -n kube-system | less
Take a look: service-pod-range, changing pod range, changing-nodeport-range.

how to change the port of a kubernetes container/pod?

I am displaying the output of the "docker ps -a command" to list all the containers to my Html page. I want to change the port of these containers using a button in the page itself. In docker normally if the container is running, I would run a docker stop on the container-id and restart it by adding the -p HOSTPORT:CONTAINERPORT to the command. But since all the containers running are Kubernetes containers/pods, stopping them will re-create a new pod/container with a different name. So how do I change the port of the container/pod in such cases?
output of "docker ps -a command"
NAMES CONTAINER ID STATUS
k8s_nginx_nginx-6cdb6c86d4-z7m7m 56711e6de1be Up 2 seconds
k8s_POD_nginx-6cdb6c86d4-z7m7m_d 70b21761cb74 Up 3 seconds
k8s_coredns_coredns-5c98db65d4-7 dfb21bb7c7f4 Up 7 days
k8s_POD_coredns-5c98db65d4-7djs8 a336be8230ce Up 7 days
k8s_POD_kube-proxy-9722h_kube-sy 5e290420dec4 Up 7 days
k8s_POD_kube-apiserver-wootz_kub a23dea72b38b Exited (255) 7 days ago
nginx.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
type: NodePort
ports:
- name: nginxport
port: 80
targetPort: 80
nodePort: 30000
selector:
app: nginx
tier: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
tier: frontend
template:
metadata:
labels:
app: nginx
tier: frontend
spec:
containers:
- image: suji165475/devops-sample:mxgraph
name: nginx
ports:
- containerPort: 80
name: nginxport
So how can I change the port of any of the containers/pod ?
Most of the attributes of a PodSpec cannot be changed once the pod has been created. The port information is inside the containers array, and the linked documentation explicitly notes that containers "Cannot be updated." You must delete and recreate the pod if you want to change the ports it makes visible (or most of its other properties); there is no other way to do it.
You almost never directly deal with Pods (and for that matter you almost never mix plain Docker containers and Kubernetes on the same host). Typically you create a Deployment object, which can be updated in place, and it takes responsibility for creating and deleting Pods for you.
(The corollary to this is that if you're trying to manually delete and recreate Pods, in isolation, changing their properties, but these Pods are also managed by Deployments or StatefulSets or DaemonSets, the controller will notice that a replica is missing when you delete it and recreate it, with its original settings.)
Answering OP's question, as per his comments.
I want to change the port on which my kubernetes containers run. I want to change the nodeport,container port,targetport for it. so how can do this using kubectl patch command for both the service and deployment?
kubectl patch deployment nginx --type json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/ports/0/containerPort", "new port"}]' && \
kubectl patch service nginx --type json -p='[{"op": "replace", "path": "/spec/type/spec/ports/0/targetPort", "new port"}]' && \
kubectl patch service nginx --type json -p='[{"op": "replace", "path": "/spec/type/spec/ports/0/nodePort", "new port"}]'
Here is how to change pod specs,
kubectl patch pod valid-pod --type='json' -p='[{"op": "replace", "path": "/spec/containers/0/ports/0/port", "value":"new port"}]'
As David said, Pods aren't really used directly without a deployment.
What you would normally do, have a deployment with deploys the pods and that configuration can be then edited using kubectl.
Try using something like this,
kubectl patch deployment valid-deployment --type json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/ports/0/port", "new port"}]'
If you patch the deployment, the pods automatically restart.
That being said, if you change the port of the container, the service targetport would have to be changed too. The simple fix for that would to make sure all your container ports have name attribute filled which are mapped to their appropriate k8s services.

How to mount HostPath Volume in Kubernetes with SELinux

I am trying to mount a hostPath volume into a Kubernetes Pod. An example of a hostPath volume specification is shown below, which is taken from the docs. I am deploying to hosts that are running RHEL 7 with SELinux enabled.
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
When my Pod tries to read from a file that has been mounted from the underlying host, I get a "Permission Denied" error. When I run setenforce 0 to turn off SELinux, the error goes away and I can access the file. I get the same error when I bind mount a directory into a Docker container.
The issue is described here and, when using Docker, can be fixed by using the z or Z bind mount flag, described in the Docker docs here.
Whilst I can fix the issue by running
chcon -Rt svirt_sandbox_file_t /path/to/my/host/dir/to/mount
I see this as a nasty hack, as I need to do this on every host in my Kubernetes cluster and also because my deployment of Kubernetes as described in the YAML spec is not a complete description of what it is that needs to be done to get my YAML to run correctly. Turning off SELinux is not an option.
I can see that Kubernetes mentions SELinux security contexts in the docs here, but I haven't been able to successfully mount a hostPath volume into a pod without getting the permission denied error.
What does the YAML need to look like to successfully enable a container to mount a HostPath volume from an underlying host that is running SELinux?
Update:
The file I am accessing is a CA certificate that has these labels:
system_u:object_r:cert_t:s0
When I use the following options:
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
and then check the access control audit errors via ausearch -m avc -ts recent, I can see that there is a permission denied error where the container has a level label of s0:c123,c456, so I can see that the level label works. I have set the label to be s0.
However, if I try to change the type label to be cert_t, the container doesn't even start, there's an error :
container_linux.go:247: starting container process caused "process_linux.go:364: container init caused \"write /proc/self/task/1/attr/exec: invalid argument\""
I don't seem to be able to change the type label of the container.
Expanding on the answer from VAS as it is partially correct:
You can only specify the level portion of an SELinux label when relabeling a path destination pointed to by a hostPath volume. This is automatically done so by the seLinuxOptions.level attribute specified in your securityContext.
However attributes such as seLinuxOptions.type currently have no effect on volume relabeling. As of this writing, this is still an open issue within Kubernetes
You can assign SELinux labels using seLinuxOptions:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
securityContext:
seLinuxOptions: # it may don’t have the desired effect
level: "s0:c123,c456"
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
# this field is optional
type: Directory
According to documentation:
Thanks to Phil for pointing that out. It appears to be working only in Pod.spec.securityContext according to the issue comment
seLinuxOptions: Volumes that support SELinux labeling are relabeled to be accessible by the label specified under seLinuxOptions. Usually you only need to set the level section. This sets the Multi-Category Security (MCS) label given to all Containers in the Pod as well as the Volumes.
You could try with full permissions:
...
image: k8s.gcr.io/test-webserver
securityContext:
privileged: true
...
Using selinux can solve this problem. Reference article:
https://zhimin-wen.medium.com/selinux-policy-for-openshift-containers-40baa1c86aa5
In addition: You can refer to the selinux parameters to set the addition, deletion, and modification of the mount directory
https://selinuxproject.org/page /ObjectClassesPerms
my setting:
If a directory's selinux is unconfined_u:object_r:kubernetes_file_t:s0 , you kan defind a selinux policy is:
module myapp 1.0;
require {
type kubernetes_file_t;
type container_t;
class file { create open read unlink write getattr execute setattr link };
class dir { add_name create read remove_name write };
}
#============= container_t ==============
#!!!! This avc is allowed in the current policy
allow container_t kubernetes_file_t:dir { add_name create read remove_name write };
#!!!! This avc is allowed in the current policy
allow container_t kubernetes_file_t:file { create open read unlink write getattr execute setattr link };
run command on node:
sudo checkmodule -M -m -o myapp.mod myapp.te
sudo semodule_package -o myapp.pp -m myapp.mod
sudo semodule -i myapp.pp

distribute docker containers evenly with kubectl

If I create 3 nodes in a cluster, how do I distribute the docker containers evenly across the containers? For example, if I create a cluster of 3 nodes with 8 cpus on each node, I've determined through performance profiling that I get the best performance when I run one container per cpu.
gcloud container clusters create mycluster --num-nodes 3 --machine-type n1-standard-8
kubectl run myapp --image=gcr.io/myproject/myapp -r 24
When I ran kubectl above, it put 11 containers on the first node, 10 on the second, and 3 on the third. How to I make it so that it is 8 each?
Both your and jpapejr's solutions seem like they'd work, but using a nodeSelector to force scheduling to a single node has the downside of requiring multiple RCs for a single application and making that application less resilient to a node failure. The idea of a custom scheduler is nice but has the downside of the amount of work to write and maintain that code.
I think another possible solution would be to set runtime constraints in your pod spec that might get you near to what you want. Based on this newly merged doc with examples of runtime contraints, I think you could set resources.requests.cpu in the pod spec part of the RC and get close to a CPU-per-pod:
apiVersion: v1
kind: Pod
metadata:
name: myapp
spec:
containers:
- name: myapp
image: myregistry/myapp:v1
resources:
requests:
cpu: "1000m"
That docs has other good examples of how requests and limits differ and interact. There may be a combination that gives you what you want and also keeps your application at proper capacity when an individual node fails.
If I'm not mistaken, what you see is the expectation. If you want finer grained control over pod placement you probably want a customer scheduler.
In my case, I want to put a fixed number of containers in each node. I am able to do this by labeling each node and then using a nodeSelector with a config. Ignore that fact that I mislabeled the 3rd node, here is my setup:
kubectl label nodes gke-n3c8-7d9f8163-node-dol5 node=1
kubectl label nodes gke-n3c8-7d9f8163-node-hmbh node=2
kubectl label nodes gke-n3c8-7d9f8163-node-kdc4 node=3
That can be automated doing:
kubectl get nodes --no-headers | awk '{print NR " " $1}' | xargs -l bash -c 'kubectl label nodes $1 node=$0'
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 8
selector:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
nodeSelector:
node: "1"
containers:
- name: nginx
image: nginx

Pulling images from private registry in Kubernetes

I have built a 4 node kubernetes cluster running multi-container pods all running on CoreOS. The images come from public and private repositories. Right now I have to log into each node and manually pull down the images each time I update them. I would like be able to pull them automatically.
I have tried running docker login on each server and putting the .dockercfg file in /root and /core
I have also done the above with the .docker/config.json
I have added secret to the kube master and added imagePullSecrets:
name: docker.io to the Pod configuration file.
When I create the pod i get the error message Error:
image <user/image>:latest not found
If I log in and run docker pull it will pull the image. I have tried this using docker.io and quay.io.
To add to what #rob said, as of docker 1.7, the use of .dockercfg has been deprecated and they now use a ~/.docker/config.json file. There is support for this type of secret in kube 1.1, but you must create it using different keys/type configuration in the yaml:
First, base64 encode your ~/.docker/config.json:
cat ~/.docker/config.json | base64 -w0
Note that the base64 encoding should appear on a single line so with -w0 we disable the wrapping.
Next, create a yaml file:
my-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: registrypullsecret
data:
.dockerconfigjson: <base-64-encoded-json-here>
type: kubernetes.io/dockerconfigjson
-
$ kubectl create -f my-secret.yaml && kubectl get secrets
NAME TYPE DATA
default-token-olob7 kubernetes.io/service-account-token 2
registrypullsecret kubernetes.io/dockerconfigjson 1
Then, in your pod's yaml you need to reference registrypullsecret or create a replication controller:
apiVersion: v1
kind: Pod
metadata:
name: my-private-pod
spec:
containers:
- name: private
image: yourusername/privateimage:version
imagePullSecrets:
- name: registrypullsecret
If you need to pull an image from a private Docker Hub repository, you can use the following.
Create your secret key
kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
secret "myregistrykey" created.
Then add the newly created key to your Kubernetes service account.
Retrieve the current service account
kubectl get serviceaccounts default -o yaml > ./sa.yaml
Edit sa.yaml and add the ImagePullSecret after Secrets
imagePullSecrets:
- name: myregistrykey
Update the service account
kubectl replace serviceaccount default -f ./sa.yaml
I can confirm that imagePullSecrets not working with deployment, but you can
kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
kubectl edit serviceaccounts default
Add
imagePullSecrets:
- name: myregistrykey
To the end after Secrets, save and exit.
And its works. Tested with Kubernetes 1.6.7
Kubernetes supports a special type of secret that you can create that will be used to fetch images for your pods. More details here.
For centos7, the docker config file is under /root/.dockercfg
echo $(cat /root/.dockercfg) | base64 -w 0
Copy and paste result to secret YAML based on the old format:
apiVersion: v1
kind: Secret
metadata:
name: docker-secret
type: kubernetes.io/dockercfg
data:
.dockercfg: <YOUR_BASE64_JSON_HERE>
And it worked for me, hope that could also help.
The easiest way to create the secret with the same credentials that your docker configuration is with:
kubectl create secret generic myregistry --from-file=.dockerconfigjson=$HOME/.docker/config.json
This already encodes data in base64.
If you can download the images with docker, then kubernetes should be able to download them too. But it is required to add this to your kubernetes objects:
spec:
template:
spec:
imagePullSecrets:
- name: myregistry
containers:
# ...
Where myregistry is the name given in the previous command.
go the easy way, do not forget to define --type and add it to proper namespace
kubectl create secret generic YOURS-SECRET-NAME \
--from-file=.dockerconfigjson=$HOME/.docker/config.json \
--type=kubernetes.io/dockerconfigjson

Resources