I'd like to set logging level in my Wildfly Swarm app build on top of registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift image. The way to configure the app is through env var JAVA_OPTIONS, therefore I'd like to use
env:
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: my.configmap
key: log.level
- name: JAVA_OPTIONS
value: -Dswarm.logging=$LOG_LEVEL
Regrettably (but not really to my surprise) this doesn't work, the LOG_LEVEL is not resolved.
Is there a way to compose env vars from config map or do I have to modify the image to consume the env vars directly?
You can reference previously defined environment variables using the following syntax:
env:
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: my.configmap
key: log.level
- name: JAVA_OPTIONS
value: -Dswarm.logging=$(LOG_LEVEL)
Not sure that this will work for java options on startup however, I have usually edited the image and called an entrypoint.sh file where some logic would exist but its worth a try I guess.
Related
Assume there is a backend application with a private key stored in a .env file.
For the project file structure:
|-App files
|-Dockerfile
|-.env
If I run the docker image locally, the application can be reached normally by using a valid public key during the API request. However, if I deploy the container into AKS cluster by using same docker image, the application failed.
I am wondering how the container in a AKS cluster handle the .env file. What should I do to solve this problem?
Moving this out of comments for better visibility.
First and most important is docker is not the same as kubernetes. What works on docker, won't work directly on kubernetes. Docker is a container runtime, while kubernetes is a container orchestration tool which sits on top of docker (not always docker now, containerd is used as well).
There are many resources on the internet which describe the key difference. For example this one is from microsoft docs
First configmaps and secrets should be created:
Creating and managing configmaps and creating and managing secrets
There are different types of secrets which can be created.
Use configmaps/secrets as environment variables.
Further referring to configMaps and secrets as environment variables looks like (configmaps and secrets have the same structure):
apiVersion: v1
kind: Pod
metadata:
name: pod-example
spec:
containers:
- ...
env:
-
name: ADMIN_PASS
valueFrom:
secretKeyRef: # here secretref is used for sensitive data
key: admin
name: admin-password
-
name: MYSQL_DB_STRING
valueFrom:
configMapKeyRef: # this is not sensitive data so can be used configmap
key: db_config
name: connection_string
...
Use configmaps/secrets as volumes (it will be presented as file).
Below the example of using secrets as files mounted in a specific directory:
apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
containers:
- ...
volumeMounts:
- name: secrets-files
mountPath: "/mnt/secret.file1" # "secret.file1" file will be created in "/mnt" directory
subPath: secret.file1
volumes:
- name: secrets-files
secret:
secretName: my-secret # name of the Secret
There's a good article which explains and shows use cases of secrets as well as its limitations e.g. size is limited to 1Mb.
I'm currently working on project which is using Kubernetes pods to deploy applications on local machine. To avoid traffic on the Development server DB, I am migrating that process on local development environment. For this I am using Liquibase to allow DB changelogs to automatically apply once the kube pods are initialized.
I already have a mssql pod running on my cluster which consists of the target Database I want to update.
This is my docker file which contains the liquibase image, the credentials are same as my mssql pod.
FROM liquibase/liquibase
ENV URL=jdbc:sqlserver://localhost:1433;database=PlayerDB
ENV USERNAME=sa
ENV PASSWORD=pl#y123
ENV CHANGELOGFILE=changelog.xml
CMD ["sh", "-c", "docker-entrypoint.sh --url=${URL} --username=${USERNAME} --password=${PASSWORD} --classpath=/liquibase/changelog --changeLogFile=${CHANGELOGFILE} update"]
This is my changelog file which contains the sql statements I want to trigger on my DB inside mssql pod
change-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: player-changelog-v1
data:
changelog.sql: |-
--liquibase formatted sql
--changeset play:1
--Database: PlayerDB
CREATE TABLE test_table (test_id INT, test_column VARCHAR, PRIMARY KEY (test_id));
Finally I have created a job which intends to update the Database by creating the new table
apiVersion: batch/v1
kind: Job
metadata:
name: liquibase-job-v1
spec:
template:
spec:
containers:
- name: liquibase
image: play/liquibase
env:
- name: URL
value: jdbc:sqlserver://localhost:1433;database=PlayerDB
- name: USERNAME
value: sa
- name: PASSWORD
value: pl#y123
- name: CHANGELOGFILE
value: changelog.sql
volumeMounts:
- name: config-vol
mountPath: /liquibase/changelog
restartPolicy: Never
volumes:
- name: config-vol
configMap:
name: player-changelog-v1
The pod runs and then terminates on it's own but the table is not getting created.
If you logs are giving out the Starting Liquibase Liquibase Community 3.8.0 by Datical Errors: The option --url is required. The option --changeLogFile is required. then why don't you define parameters in the commands as well? They are mandatory Liquibase parameters to run the liquibase updates
I'm using this Splunk image on Kubernetes (testing locally with minikube).
After applying the code below I'm facing the following error:
ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe
$SPLUNK_HOME or $SPLUNK_ETC is set wrong?
My Splunk deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: splunk
labels:
app: splunk-app
tier: splunk
spec:
selector:
matchLabels:
app: splunk-app
track: stable
replicas: 1
template:
metadata:
labels:
app: splunk-app
tier: splunk
track: stable
spec:
volumes:
- name: configmap-inputs
configMap:
name: splunk-config
containers:
- name: splunk-client
image: splunk/splunk:latest
imagePullPolicy: Always
env:
- name: SPLUNK_START_ARGS
value: --accept-license --answer-yes
- name: SPLUNK_USER
value: root
- name: SPLUNK_PASSWORD
value: changeme
- name: SPLUNK_FORWARD_SERVER
value: splunk-receiver:9997
ports:
- name: incoming-logs
containerPort: 514
volumeMounts:
- name: configmap-inputs
mountPath: /opt/splunk/etc/system/local/inputs.conf
subPath: "inputs.conf"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: splunk-config
data:
inputs.conf: |
[monitor:///opt/splunk/var/log/syslog-logs]
disabled = 0
index=my-index
I tried to add also this env variables - with no success:
- name: SPLUNK_HOME
value: /opt/splunk
- name: SPLUNK_ETC
value: /opt/splunk/etc
I've tested the image with the following docker configuration - and it ran successfully:
version: '3.2'
services:
splunk-forwarder:
hostname: splunk-client
image: splunk/splunk:latest
environment:
SPLUNK_START_ARGS: --accept-license --answer-yes
SPLUNK_USER: root
SPLUNK_PASSWORD: changeme
ports:
- "8089:8089"
- "9997:9997"
Saw this on Splunk forum but the answer did not help in my case.
Any ideas?
Edit #1:
Minikube version: Upgraded fromv0.33.1 to v1.2.0.
Full error log:
$kubectl logs -l tier=splunk
splunk_common : Set first run fact -------------------------------------- 0.04s
splunk_common : Set privilege escalation user --------------------------- 0.04s
splunk_common : Set current version fact -------------------------------- 0.04s
splunk_common : Set splunk install fact --------------------------------- 0.04s
splunk_common : Set docker fact ----------------------------------------- 0.04s
Execute pre-setup playbooks --------------------------------------------- 0.04s
splunk_common : Setting upgrade fact ------------------------------------ 0.04s
splunk_common : Set target version fact --------------------------------- 0.04s
Determine captaincy ----------------------------------------------------- 0.04s
ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe $SPLUNK_HOME or $SPLUNK_ETC is set wrong?
Edit #2: Adding config map to the code (was removed from the original question for the sake of brevity). This is the cause of failure.
Based on the direction pointed out by #Amit-Kumar-Gupta I'll try also to give a full solution.
So this PR change makes it so that containers cannot write to secret, configMap, downwardAPI and projected volumes since the runtime will now mount them as read-only.
This change is since v1.9.4 and can lead to issues for various applications which chown or otherwise manipulate their configs.
When Splunk boots, it registers all the config files in various locations on the filesystem under ${SPLUNK_HOME} which is in our case /opt/splunk.
The error specified in the my question reflect that splunk failed to manipulate all the relevant files in the /opt/splunk/etc directory because of the change in the mounting mechanism.
Now for the solution.
Instead of mounting the configuration file directly inside the /opt/splunk/etc directory we'll use the following setup:
We'll start the docker container with a default.yml file which will be mounted in /tmp/defaults/default.yml.
For that, we'll create the default.yml file with: docker run splunk/splunk:latest create-defaults > ./default.yml
Then, We'll go to the splunk: block and add a config: sub block under it:
splunk:
conf:
inputs:
directory: /opt/splunk/etc/system/local
content:
monitor:///opt/splunk/var/log/syslog-logs:
disabled : 0
index : syslog-index
outputs:
directory: /opt/splunk/etc/system/local
content:
tcpout:splunk-indexer:
server: splunk-indexer:9997
This setup will generate two files with a .conf postfix (Remember that the sub block start with conf:) which be owned by the correct Splunk user and group.
The inputs: section will produce the a inputs.conf with the following content:
[monitor:///opt/splunk/var/log/syslog-logs]
disabled = 0
index=syslog-index
In a similar way, the outputs: block will resemble the following:
[tcpout:splunk-receiver]
server=splunk-receiver:9997
This is instead of the passing an environment variable directly like I did in the origin code:
SPLUNK_FORWARD_SERVER: splunk-receiver:9997
Now everything is up and running (:
Full setup of the forwarder.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: splunk-forwarder
labels:
app: splunk-forwarder-app
tier: splunk
spec:
selector:
matchLabels:
app: splunk-forwarder-app
track: stable
replicas: 1
template:
metadata:
labels:
app: splunk-forwarder-app
tier: splunk
track: stable
spec:
volumes:
- name: configmap-forwarder
configMap:
name: splunk-forwarder-config
containers:
- name: splunk-forwarder
image: splunk/splunk:latest
imagePullPolicy : Always
env:
- name: SPLUNK_START_ARGS
value: --accept-license --answer-yes
- name: SPLUNK_PASSWORD
valueFrom:
secretKeyRef:
name: splunk-secret
key: password
volumeMounts:
- name: configmap-forwarder
mountPath: /tmp/defaults/default.yml
subPath: "default.yml"
For further reading:
https://splunk.github.io/docker-splunk/ADVANCED.html
https://github.com/splunk/docker-splunk/blob/develop/docs/ADVANCED.md
https://www.splunk.com/blog/2018/12/17/deploy-splunk-enterprise-on-kubernetes-splunk-connect-for-kubernetes-and-splunk-insights-for-containers-beta-part-1.html
https://splunk.github.io/splunk-ansible/ADVANCED.html#inventory-script
https://static.rainfocus.com/splunk/splunkconf18/sess/1521146368312001VwQc/finalPDF/FN1089_DockerizingSplunkatScale_Final_1538666172485001Loc0.pdf
There are two questions here: (1) why are you seeing that error message, and (2) how to achieve the desired behaviour you're hoping to achieve that you're trying to express through your Deployment and ConfigMap. Unfortunately, I don't believe there's a "cloud-native" way to achieve what you want, but I can explain (1), why it's hard to do (2), and point you to something that might give you a workaround.
The error message:
ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe $SPLUNK_HOME or $SPLUNK_ETC is set wrong?
does not imply that you've set those environment variables incorrectly (necessarily), it implies that Splunk is looking for a file in that location and can't read a file there, and it's providing a hint that maybe you've put the file in another place but forgot to give Splunk the hint (via the $SPLUNK_HOME or $SPLUNK_ETC environment variables) to look elsewhere.
The reason why it can't read /opt/splunk/etc/splunk-launch.conf is because, by default, the /opt/splunk directory would be populated with tons of subdirectories and files with various configurations, but because you're mounting a volume at /opt/splunk/etc/system/local/inputs.conf, nothing can be written to /opt/splunk.
If you simply don't mount that volume, or mount it somewhere else (e.g. /foo/inputs.conf) the Deployment will start fine. Of course the problem is that it won't know anything about your inputs.conf, and it'll use the default /opt/splunk/etc/system/local/inputs.conf it writes there.
I assume what you want to do is allow Splunk to generate all the directories and files it likes, you only want to set the contents of that one file. While there is a lot of nuance about how Kubernetes deals with volume mounts, in particular those coming from ConfigMaps, and in particular when using subPath, at the end of the day I don't think there's a clean way to do what you want.
I did an Internet search for "splunk kubernetes inputs.conf" and this was my first result: https://www.splunk.com/blog/2019/02/11/deploy-splunk-enterprise-on-kubernetes-splunk-connect-for-kubernetes-and-splunk-insights-for-containers-beta-part-2.html. This is from official splunk.com, and it's advising running things like kubectl cp and kubectl exec to:
"Exec" into the master pod, and run ... commands, to copy (configuration) into the (target) directory and chown to splunk user.
🤷🏾♂️
One solution that worked for me in K8s deployment was:
Ammend below to the image Dockerfile
#RUN chmod -R 755 /opt/ansible
#RUN echo " ignore_errors: yes" >> /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml
Then use that same image in your deployment from your private repo with belo env variables:
#has to run as root otherwise won't let you write to $SPLUNK_HOME/S
env:
- name: SPLUNK_START_ARGS
value: --accept-license --answer-yes --no-prompt
- name: SPLUNK_USER
value: root
I tried to set variables in Azure Pipelines' Release which can be used Command Task in Release to replace variables' values to Docker Kubernetes' .yaml file.
It works fine for me but I need to prepare several Command Tasks to replace variables one by one.
For example, I set variable TESTING1_(value:Test1), TESTING2_(value:Test2) and TESTING3_(value:Test3) in Pipelines' Release. Then I only used Command Task to replace TESTING1_ to $(TESTING1_) in Docker Kubernetes' .yaml file. Below is original environment setting in .yaml file:
spec:
containers:
- name: devops
env:
- name: TESTING1
value: TESTING1_
- name: TESTING2
value: $(TESTING2_)
After ran Pipelines' Release, print out results in NodeJS were:
console.log(process.env.TESTING1); --> Test1
console.log(process.env.TESTING2); --> $(TESTING2_)
console.log(process.env.TESTING3); --> undefined
i think you should be use config maps for that (maybe update values in config maps). you shouldn't be updating containers directly. this gives you flexibility and management. example:
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: SPECIAL_LEVEL_KEY
valueFrom:
configMapKeyRef:
name: special-config
key: special.how
and then if some value changes you update the config map and all the pods that reference this config map get new value.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-container-environment-variables-using-configmap-data
I want to pass some values from Kubernetes yaml file to the containers. These values will be read in my Java app using System.getenv("x_slave_host").
I have this dockerfile:
FROM jetty:9.4
...
ARG slave_host
ENV x_slave_host $slave_host
...
$JETTY_HOME/start.jar -Djetty.port=9090
The kubernetes yaml file contains this part where I added env section:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: master
spec:
template:
metadata:
labels:
app: master
spec:
volumes:
- name: shared-data
emptyDir: {}
containers:
- name: master
image: xregistry.azurecr.io/Y:latest
ports:
- containerPort: 9090
volumeMounts:
- name: shared-data
mountPath: ~/.X/experiment
- env:
- name: slave_host
value: slavevalue
- name: jupyter
image: xregistry.azurecr.io/X:latest
ports:
- containerPort: 8000
- containerPort: 8888
volumeMounts:
- name: shared-data
mountPath: /var/folder/experiment
imagePullSecrets:
- name: acr-auth
Locally when I did the same thing using docker compose, it worked using args. This is a snippet:
master:
image: master
build:
context: ./master
args:
- slave_host=slavevalue
ports:
- "9090:9090"
So now I am trying to do the same thing but in Kubernetes. However, I am getting the following error (deploying it on Azure):
error: error validating "D:\\a\\r1\\a\\_X\\deployment\\kub-deploy.yaml": error validating data: field spec.template.spec.containers[1].name for v1.Container is required; if you choose to ignore these errors, turn validation off with --validate=false
In other words, how to rewrite my docker compose file to kubernetes and passing this argument.
Thanks!
env section should be added under containers, like this:
containers:
- name: master
env:
- name: slave_host
value: slavevalue
To elaborate a on #Kun Li's answer, besides adding environment variables e.g. in the Deployment manifest directly you can create a ConfigMap (or Secret depending on the data being stored) and reference these in your manifests. This is a good way of sharing the same environment variables across applications, compared to manually adding environment variables to several different applications.
Note that a ConfigMap can consist of one or more key: value pairs and it's not limited to storing environment variables, it's just one of the use cases. And as i mentioned before, consider using a Secret if the data is classified as sensitive.
Example of a ConfigMap manifest, in this case used for storing an environment variable:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-env-var
data:
slave_host: slavevalue
To create a ConfigMap holding one key=value pair using kubectl create:
kubectl create configmap my-env --from-literal=slave_host=slavevalue
To get hold of all environment variables configured in a ConfigMap use the following in your manifest:
containers:
envFrom:
- configMapRef:
name: my-env-var
Or if you want to pick one specific environment variable from your ConfigMap containing several variables:
containers:
env:
- name: slave_host
valueFrom:
configMapKeyRef:
name: my-env-var
key: slave_host
See this page for more examples of using ConfigMap's in different situations.