Kubernetes env variable to containers - docker

I want to pass some values from Kubernetes yaml file to the containers. These values will be read in my Java app using System.getenv("x_slave_host").
I have this dockerfile:
FROM jetty:9.4
...
ARG slave_host
ENV x_slave_host $slave_host
...
$JETTY_HOME/start.jar -Djetty.port=9090
The kubernetes yaml file contains this part where I added env section:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: master
spec:
template:
metadata:
labels:
app: master
spec:
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: master
image: xregistry.azurecr.io/Y:latest
ports:
- containerPort: 9090
volumeMounts:
- name: shared-data
mountPath: ~/.X/experiment
- env:
- name: slave_host
value: slavevalue
- name: jupyter
image: xregistry.azurecr.io/X:latest
ports:
- containerPort: 8000
- containerPort: 8888
volumeMounts:
- name: shared-data
mountPath: /var/folder/experiment
imagePullSecrets:
- name: acr-auth
Locally when I did the same thing using docker compose, it worked using args. This is a snippet:
master:
image: master
build:
context: ./master
args:
- slave_host=slavevalue
ports:
- "9090:9090"
So now I am trying to do the same thing but in Kubernetes. However, I am getting the following error (deploying it on Azure):
error: error validating "D:\\a\\r1\\a\\_X\\deployment\\kub-deploy.yaml": error validating data: field spec.template.spec.containers[1].name for v1.Container is required; if you choose to ignore these errors, turn validation off with --validate=false
In other words, how to rewrite my docker compose file to kubernetes and passing this argument.
Thanks!

env section should be added under containers, like this:
containers:
- name: master
env:
- name: slave_host
value: slavevalue

To elaborate a on #Kun Li's answer, besides adding environment variables e.g. in the Deployment manifest directly you can create a ConfigMap (or Secret depending on the data being stored) and reference these in your manifests. This is a good way of sharing the same environment variables across applications, compared to manually adding environment variables to several different applications.
Note that a ConfigMap can consist of one or more key: value pairs and it's not limited to storing environment variables, it's just one of the use cases. And as i mentioned before, consider using a Secret if the data is classified as sensitive.
Example of a ConfigMap manifest, in this case used for storing an environment variable:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-env-var
data:
slave_host: slavevalue
To create a ConfigMap holding one key=value pair using kubectl create:
kubectl create configmap my-env --from-literal=slave_host=slavevalue
To get hold of all environment variables configured in a ConfigMap use the following in your manifest:
containers:
envFrom:
- configMapRef:
name: my-env-var
Or if you want to pick one specific environment variable from your ConfigMap containing several variables:
containers:
env:
- name: slave_host
valueFrom:
configMapKeyRef:
name: my-env-var
key: slave_host
See this page for more examples of using ConfigMap's in different situations.

Related

Checkpoint pushgateway persistence file to object store

I am using pushgateway to exposes metrics coming from short-lived batch jobs.
At the moment the pushgateway instance is launched on a baremetal machine, where I have a docker volume mounted to allow survival of metrics in case of a container restart (in conjunction with the --persistence.file parameter).
Here an extract of the docker-compose.yml file used to run the container:
pushgateway:
image: prom/pushgateway:v1.2.0
restart: unless-stopped
volumes:
- pushgw-data:/data
ports:
- "${PUSHGW_PORT:-9091}:9091"
command: --persistence.file="/data/metric.store"
I am moving to a (private) kubernetes cluster without persistent volumes, but equipped with an s3-compatible object storage.
From this issue on github it seems possible to target s3 for the checkpointing, but without further input I am not sure how to achieve this, and that's the best I could find by searching the Web for information.
Can anyone point me in the right direction?
So finally https://serverfault.com/questions/976764/kubernetes-run-aws-s3-sync-rsync-against-persistent-volume-on-demand pointed me in the right direction.
This is an extract of the deployment.yaml descriptor which works as expected:
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: {{K8S_NAMESPACE}}
name: {{K8S_DEPLOYMENT_NAME}}
spec:
selector:
matchLabels:
name: {{K8S_DEPLOYMENT_NAME}}
strategy:
type: Recreate
template:
metadata:
labels:
name: {{K8S_DEPLOYMENT_NAME}}
version: v1
spec:
containers:
- name: {{AWSCLI_NAME}}
image: {{IMAGE_AWSCLI}}
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{SECRET_NAME}}
key: accesskey
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{SECRET_NAME}}
key: secretkey
command: [ "/bin/bash",
"-c",
"aws --endpoint-url {{ENDPOINT_URL}} s3 sync s3://{{BUCKET}} /data; while true; do aws --endpoint-url {{ENDPOINT_URL}} s3 sync /data s3://{{BUCKET}}; sleep 60; done" ]
volumeMounts:
- name: pushgw-data
mountPath: /data
- name: {{PUSHGATEWAY_NAME}}
image: {{IMAGE_PUSHGATEWAY}}
command: [ '/bin/sh', '-c' ]
args: [ 'sleep 10; /bin/pushgateway --persistence.file=/data/metric.store' ]
ports:
- containerPort: 9091
volumeMounts:
- name: pushgw-data
mountPath: /data
volumes:
- name: pushgw-data
emptyDir: {}
- name: config-volume
configMap:
name: {{K8S_DEPLOYMENT_NAME}}
imagePullSecrets:
- name: harbor-bot
restartPolicy: Always
Note the override of entrypoint for the docker image of the pushgateway. In my case I have put 10 seconds delay to start, you might need to tune the delay to suits your needs. This delay is needed because the pushgateway container will boot faster than the sidecar (also due to the network exchange with s3, I suppose).
If the pushgateway starts when not metric store file is already present, it won't be used/considered. But it gets worse, when you first send data to the pushgateway, it will override the file. At that point, the "sync" from the sidecar container will also override the original "copy", so please pay attention and be sure you have a backup of the metrics file before experimenting with this delay value.

Copy files into kubernetes pod with deployment.yaml

I have containerized microservice built with Java. This application uses the default /config-volume directory when it searches for property files.
Previously I manually deployed via Dockerfile, and now I'm looking to automate this process with Kubernetes.
The container image starts the microservice immediately so I need to add properties to the config-volume folder immediately. I accomplished this in Docker with this simple Dockerfile:
FROM ########.amazon.ecr.url.us-north-1.amazonaws.com/company/image-name:1.0.0
RUN mkdir /config-volume
COPY path/to/my.properties /config-volume
I'm trying to replicate this type of behavior in a kubernetes deployment.yaml but I have found no way to do it.
I've tried performing a kubectl cp command immediately after applying the deployment and it sometimes works, but it can result in a race condition which cause the microservice to fail at startup.
(I've redacted unnecessary parts)
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-service
spec:
replicas: 1
template:
spec:
containers:
- env:
image: ########.amazon.ecr.url.us-north-1.amazonaws.com/company/image-name:1.0.0
name: my-service
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /config-volume
name: config-volume
volumes:
- name: config-volume
emptyDir: {}
status: {}
Is there a way to copy files into a volume inside the deployment.yaml?
You are trying to emulate a ConfigMap using volumes. Instead, put your configuration into a ConfigMap, and mount that to your deployments. The documentation is there:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
Once you have your configuration as a ConfigMap, mount it using something like this:
...
containers:
- name: mycontainer
volumeMounts:
- name: config-volume
mountPath: /config-volume
volumes:
- name: config-volume
configMap:
name: nameOfConfigMap

env vars in postgres deployment not working

First off, I'm pretty sure I know why this isn't working: I'm pulling the Docker postgres:11-alpine image, modifying it, but then trying to change the env: in the k8s deployment.yaml on a custom image. I think that is the issue.
Basically, I'm trying to accomplish this per the Docker postgres docs:
docker run --name some-postgres -e POSTGRES_PASSWORD='foo' POSTGRES_USER='bar'
This is what I have:
Dockerfile.dev
FROM postgres:11-alpine
EXPOSE 5432
COPY ./db/*.sql /docker-entrypoint-initdb.d/
postgres.yaml (secrets will be moved after I'm done playing with this)
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-deployment
spec:
replicas: 1
selector:
matchLabels:
component: postgres
template:
metadata:
labels:
component: postgres
spec:
containers:
- name: postgres
image: testproject/postgres
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: "test_dev"
- name: POSTGRES_USER
value: "bar"
- name: POSTGRES_PASSWORD
value: "foo"
volumeMounts:
- name: postgres-storage
mountPath: /var/lib/postgresql/data
subPath: postgres
volumes:
- name: postgres-storage
persistentVolumeClaim:
claimName: postgres-storage
---
apiVersion: v1
kind: Service
metadata:
name: postgres-cluster-ip-service
spec:
type: ClusterIP
selector:
component: postgres
ports:
- port: 5432
targetPort: 5432
When I use Skaffold to spin the cluster up locally, however, the env: "don't take" as I can still access the DB using the defaults POSTGRES_USER=postgres and POSTGRES_PASSWORD=''.
I bet if I did image: postgres then the env: would work, but then I'm not sure how to do the equivalent of this that is in the Dockerfile:
COPY ./db/*.sql /docker-entrypoint-initdb.d/
Any suggestions?
Here is the skaffold.yaml if that is helpful too:
apiVersion: skaffold/v1beta15
kind: Config
build:
local:
push: false
artifacts:
- image: testproject/postgres
docker:
dockerfile: ./db/Dockerfile.dev
sync:
manual:
- src: "***/*.sql"
dest: .
- image: testproject/server
docker:
dockerfile: ./server/Dockerfile.dev
sync:
manual:
- src: "***/*.py"
dest: .
deploy:
kubectl:
manifests:
- k8s/ingress.yaml
- k8s/postgres.yaml
- k8s/server.yaml
The Docker postgres docs mention the following:
Warning: the Docker specific variables will only have an effect if you start the container with a data directory that is empty; any pre-existing database will be left untouched on container startup.
Are you sure that you're starting your deployment with an empty data directory? Could it be that PostgreSQL starts and allows you to login using the credentials that were specified in the environment variables during the first time your started it with that persistent volume?
If that's not it, have a look at the environment variables of the running pod. kubectl describe POD should tell you which environment variables are actually passed through to the pod. Maybe something in your Skaffold setup overwrites the environment variables? You could have a look in the pod by running env when execing into the pod. Also don't forget the logs, the PostgreSQL container should log which user account it creates during startup.

Store/Share data with a container in Kubernetes

I've dockerized a python project that requires the use of several CSVs (~2gb). In order to keep image size down I didn't include the CSVs in the build, instead opting to give the running container the data from a directory outside the container through a volume. Locally, when running through docker, I can just do
docker run -v ~/local/path/:/container/path my-image:latest
This works, but I'm not sure how to go about doing this in Kubernetes. I've been reading the documentation and am confused by the number of volume types, where the actual CSVs should be stored, etc.
Based on the information about the project that I've provided, is there an obvious solution?
If you'd like to replicate that exact behavior from Docker the most common way to do it is to use hostPath. Something like this:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: my-image:latest
name: my-container
volumeMounts:
- mountPath: /container/path
name: test-volume
volumes:
- name: test-volume
hostPath:
path: /usr/local/path
type: Directory
Here is a typical example of sharing between containers. You can keep your data in a separate container and code in a different container.
https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/
apiVersion: v1
kind: Pod
metadata:
name: two-containers
spec:
restartPolicy: Never
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: nginx-container
image: nginx
volumeMounts:
- name: shared-data
mountPath: /usr/share/nginx/html
- name: debian-container
image: debian
volumeMounts:
- name: shared-data
mountPath: /pod-data
command: ["/bin/sh"]
args: ["-c", "echo Hello from the debian container > /pod-data/index.html"]
Hope it helps.

Is there any definitive guide on how to pass all the arguments to Docker containers while starting a container through kubernetes?

I want to start a docker container with Kubernetes with the parameter --oom-score-adj .
My kubernetes deployment script looks like this:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: xxx
spec:
template:
metadata:
labels:
app: xxx
spec:
volumes:
- name: some-name
hostPath:
path: /some-path
containers:
- name: xxx-container
image: xxx-image
imagePullPolicy: "IfNotPresent"
securityContext:
privileged: true
command:
- /bin/sh
- -c
args:
- ./rsome-command.sh
volumeMounts:
- name: some-name
mountPath: /some-path
When I inspect the created container, I find --oom-score-adj is set to 1000. I want to set it to 0. Can anyone shed any line on how can I do it? Is there any definitive guide to pass such arguments?
You can't do this yet, it's one of the frustrating things still unresolved with Kubernetes.
There's a similar issue here around logging drivers. Unfortunately, you'll have to set the value on the docker daemon

Resources