I have setup VerneMQ to pull CRL (Certificate Revocation List) from Vault. Followingis my setup manifest.
kind: ConfigMap
apiVersion: v1
metadata:
name: vernemq-refresh-crl
namespace: backend
labels:
app: vernemq
data:
pull_crl.sh: |
#!/usr/bin/env sh
if ! apk info | grep ^curl ; then apk update && apk add curl; fi
while true
do
echo $(date)
curl -w "\n" --header "X-Vault-Token: $VAULT_TOKEN" http://vault.backend.svc.cluster.local:8200/v1/my-ca/crl/pem > /tmp/shared/ca.crl
sleep $SLEEP_INTERVAL
done
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vernemq
namespace: backend
spec:
replicas: 1
selector:
matchLabels:
app: vernemq
template:
metadata:
labels:
app: vernemq
spec:
serviceAccountName: vernemq
containers:
- name: vernemq
image: vernemq/vernemq:1.12.3
ports:
- name: mqtt
containerPort: 1883
- name: mqtts
containerPort: 8883
- name: mqtt-ws
containerPort: 8080
- name: epmd
containerPort: 4369
- name: vmq
containerPort: 44053
- name: metrics
containerPort: 8888
env:
- name: DOCKER_VERNEMQ_ACCEPT_EULA
value: "yes"
- name: MY_POD_NAME
value: "vernemq"
- name: DOCKER_VERNEMQ_KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: DOCKER_VERNEMQ_KUBERNETES_APP_LABEL
value: "vernemq"
- name: DOCKER_VERNEMQ_LOG__CONSOLE__LEVEL
value: "debug"
- name: DOCKER_VERNEMQ_KUBERNETES_LABEL_SELECTOR
value: "app=vernemq"
- name: DOCKER_VERNEMQ_LISTENER__TCP__ALLOWED_PROTOCOL_VERSIONS
value: "3,4,5"
- name: DOCKER_VERNEMQ_LISTENER__SSL__ALLOWED_PROTOCOL_VERSIONS
value: "3,4,5"
- name: DOCKER_VERNEMQ_ALLOW_ANONYMOUS
value: "on"
- name: DOCKER_VERNEMQ_TOPIC_MAX_DEPTH
value: "20"
- name: DOCKER_VERNEMQ_KUBERNETES_INSECURE
value: "1"
- name: DOCKER_VERNEMQ_MAX_ONLINE_MESSAGES
value: "-1"
- name: DOCKER_VERNEMQ_MAX_OFFLINE_MESSAGES
value: "-1"
- name: DOCKER_VERNEMQ_MAX_INFLIGHT_MESSAGES
value: "0"
- name: DOCKER_VERNEMQ_LISTENER__TCP__DEFAULT
value: "0.0.0.0:1883"
- name: DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT
value: "0.0.0.0:8883"
- name: DOCKER_VERNEMQ_LISTENER__WS__DEFAULT
value: "0.0.0.0:8080"
- name: DOCKER_VERNEMQ_LISTENER__HTTP__METRICS
value: "0.0.0.0:8888"
- name: DOCKER_VERNEMQ_LISTENER__HTTP__DEFAULT
value: "0.0.0.0:8888"
- name: DOCKER_VERNEMQ_LISTENER__SSL__REQUIRE_CERTIFICATE
value: "on"
- name: DOCKER_VERNEMQ_LISTENER__SSL__USE_IDENTITY_AS_USERNAME
value: "on"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CAFILE
value: "/vernemq/cert/ca.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE
value: "/vernemq/cert/server.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE
value: "/vernemq/cert/server.key"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CRLFILE
value: "/tmp/shared/ca.crl"
volumeMounts:
- name: cert
mountPath: /vernemq/cert
readOnly: true
- name: acl
mountPath: /vernemq/acl
readOnly: true
- name: tmp-shared
mountPath: /tmp/shared
- name: pull-crl
image: alpine
command: ["/bin/sh"]
args: ["-c", "/script/pull_crl.sh"]
env:
- name: VAULT_TOKEN
valueFrom:
secretKeyRef:
name: vault
key: root-token
- name: SLEEP_INTERVAL
value: "300"
volumeMounts:
- name: pull-crl-sh
mountPath: /script/pull_crl.sh
subPath: pull_crl.sh
- name: tmp-shared
mountPath: /tmp/shared
volumes:
- name: cert
secret:
secretName: vernemq-cert
- name: pull-crl-sh
configMap:
name: vernemq-refresh-crl
defaultMode: 0744
- name: tmp-shared
emptyDir: {}
I do see serial number of revoked certificate in /tmp/shared/ca.crl but client using the revoked certificate is able to maintain already establish session with broker.
Is there a easy way to kick the client with revoked certificate
Check the administrative disconnect command with sudo vmq-admin session disconnect --help.
In general, it is a wrong assumption that adding a cert to the CRL will disrupt an existing connection. You have to couple that step with an administrative disconnect of the client.
Related
I want to configure airflow on openshift.
I set database on openshift like below :
kind: Service
apiVersion: v1
metadata:
name: airflow-database
namespace: ersin-poc
spec:
ports:
- name: 5432-tcp
protocol: TCP
port: 5432
targetPort: 5432
selector:
deployment: airflow-database
type: ClusterIP
sessionAffinity: None
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
and my database deployment like below :
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-database
namespace: ersin-poc
labels:
deployment: airflow-database
spec:
replicas: 1
selector:
matchLabels:
deployment: airflow-database
template:
metadata:
creationTimestamp: null
labels:
deployment: airflow-database
spec:
volumes:
- name: generic
persistentVolumeClaim:
claimName: generic
- name: empty1
emptyDir: {}
containers:
- resources: {}
name: airflow-database
env:
- name: POSTGRESQL_USERNAME
value: 'bn_airflow'
- name: POSTGRESQL_PASSWORD
value: 'bitnami1'
- name: POSTGRESQL_DATABASE
value: 'bitnami_airflow'
ports:
- containerPort: 5432
protocol: TCP
volumeMounts:
- name: generic
mountPath: /bitnami/postgresql/
image: >-
bitnami/postgresql:latest
hostname: airflow-database
I can connect this db from my webserver like below :
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-webserver
namespace: ersin-poc
labels:
deployment: airflow-webserver
spec:
replicas: 1
selector:
matchLabels:
deployment: airflow-webserver
template:
metadata:
creationTimestamp: null
labels:
deployment: airflow-webserver
spec:
volumes:
- name: generic
persistentVolumeClaim:
claimName: generic
- name: empty1
emptyDir: {}
containers:
- resources: {}
name: airflow-webserver
env:
- name: AIRFLOW_HOME
value: /home/appuser
- name: USER
value: appuser
- name: AIRFLOW_FERNET_KEY
value: '46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho='
- name: AIRFLOW_SECRET_KEY
value: 'a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08='
- name: AIRFLOW_EXECUTOR
value: 'CeleryExecutor'
- name: AIRFLOW_DATABASE_NAME
value: 'bitnami_airflow'
- name: AIRFLOW_DATABASE_USERNAME
value: 'bn_airflow'
- name: AIRFLOW_DATABASE_PASSWORD
value: 'bitnami1'
- name: AIRFLOW_LOAD_EXAMPLES
value: 'yes'
- name: AIRFLOW_PASSWORD
value: 'bitnami123'
- name: AIRFLOW_USERNAME
value: 'user'
- name: AIRFLOW_EMAIL
value: 'user#example.com'
- name: AIRFLOW_DATABASE_HOST
value: 'airflow-database'
- name: AIRFLOW_DATABASE_PORT_NUMBER
value: '5432'
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: generic
mountPath: /home/appuser
- name: generic
mountPath: /home/appuser/logs/
- name: generic
mountPath: /home/appuser/dags/
image: >-
bitnami/airflow:latest
hostname: airflow-webserver
but when i try it with airflow-scheduler it gives error :
airflow-scheduler 09:29:43.31 INFO ==> Trying to connect to the database server airflow-scheduler 09:30:47.42 ERROR ==> Could not connect to the database
and my scheduler yaml is :
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-scheduler
namespace: ersin-poc
labels:
deployment: airflow-scheduler
spec:
replicas: 1
selector:
matchLabels:
deployment: airflow-scheduler
template:
metadata:
labels:
deployment: airflow-scheduler
spec:
volumes:
- name: generic
persistentVolumeClaim:
claimName: generic
- name: empty1
emptyDir: {}
containers:
- resources: {}
name: airflow-scheduler
env:
- name: AIRFLOW_HOME
value: /home/appuser
- name: USER
value: appuser
- name: AIRFLOW_FERNET_KEY
value: '46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho='
- name: AIRFLOW_SECRET_KEY
value: 'a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08='
- name: AIRFLOW_EXECUTOR
value: 'CeleryExecutor'
- name: AIRFLOW_DATABASE_NAME
value: 'bitnami_airflow'
- name: AIRFLOW_DATABASE_USERNAME
value: 'bn_airflow'
- name: AIRFLOW_DATABASE_PASSWORD
value: 'bitnami1'
- name: AIRFLOW_DATABASE_HOST
value: 'airflow-database'
- name: AIRFLOW_DATABASE_PORT_NUMBER
value: '5432'
- name: AIRFLOW_WEBSERVER_HOST
value: 'airflow-webserver'
- name: AIRFLOW_WEBSERVER_PORT_NUMBER
value: '8080'
- name: REDIS_HOST
value: 'airflow-redis'
- name: REDIS_PORT_NUMBER
value: '6379'
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: generic
mountPath: /home/appuser
- name: generic
mountPath: /home/appuser/logs/
- name: generic
mountPath: /home/appuser/dags/
image: >-
bitnami/airflow-scheduler:latest
hostname: airflow-scheduler
so i cant understand why i got this error with same properties?
thanks in advance
EDIT
and I try in scheduler pod this commands to see whether i can connect to db or not :
psql -h airflow-database -p 5432 -U bn_airflow -d bitnami_airflow -W
pass: bitnami1
select * from public.ab_user;
and yes I can.
After a lot of search , I ve decided to make this with apache/airflow images. (posgresql and redis are still bitnami - it doesnt important)
You can see all ymal files about airflow on openshift.
https://github.com/ersingulbahar/airflow_on_openshift
It works now as expected
I am trying to load elasticsearch.yml file using ConfigMap while installing ElasticSearch using Kubernetes.
kubectl create configmap elastic-config --from-file=./elasticsearch.yml
The elasticsearch.yml file is loaded in the container with root as its owner and read-only permission (https://github.com/kubernetes/kubernetes/issues/62099). Since, ElasticSearch will not start with root ownership, the pod crashes.
As a work-around, I tried to mount the ConfigMap to a different file and then copy it to the config directory using an initContainer. However, the file in the config directory does not seem to be updated.
Is there anything that I am missing or is there any other way to accomplish this?
ElasticSearch Kubernetes StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-cluster
labels:
app: elasticservice
spec:
serviceName: elasticsearch
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
volumeMounts:
- name: elastic-config-vol
mountPath: /tmp/elasticsearch
- name: elastic-storage
mountPath: /usr/share/elasticsearch/data
env:
- name: cluster.name
value: docker-elastic
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.zen.ping.unicast.hosts
value: "elastic-service"
- name: discovery.zen.minimum_master_nodes
value: "1"
- name: node.master
value: "true"
- name: node.data
value: "true"
- name: ES_JAVA_OPTS
value: "-Xmx256m -Xms256m"
volumes:
- name: elastic-config-vol
configMap:
name: elastic-config
items:
- key: elasticsearch.yml
path: elasticsearch.yml
- name: elastic-config-dir
emptyDir: {}
- name: elastic-storage
emptyDir: {}
initContainers:
# elasticsearch will not run as non-root user, fix permissions
- name: fix-vol-permission
image: busybox
command:
- sh
- -c
- chown -R 1000:1000 /usr/share/elasticsearch/data
securityContext:
privileged: true
volumeMounts:
- name: elastic-storage
mountPath: /usr/share/elasticsearch/data
- name: fix-config-vol-permission
image: busybox
command:
- sh
- -c
- cp /tmp/elasticsearch/elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml
securityContext:
privileged: true
volumeMounts:
- name: elastic-config-dir
mountPath: /usr/share/elasticsearch/config
- name: elastic-config-vol
mountPath: /tmp/elasticsearch
# increase default vm.max_map_count to 262144
- name: increase-vm-max-map-count
image: busybox
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
- name: increase-the-ulimit
image: busybox
command:
- sh
- -c
- ulimit -n 65536
securityContext:
privileged: true
I use:
...
volumeMounts:
- name: config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
volumes:
- name : config
configMap:
name: es-configmap
without any permissions problem, but you can set permissions with defaultMode
I am trying to clone a private git repository(gitLab) into a kubernetes pod, using SSH keys for authentication. I have stored my keys in a secret. Here is the yaml file for the job that does the desired task.
Heres the same question, but doesnt give the exact solution :
Clone a secure git repo in Kubernetes pod
Logs of the init container after execution:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
v3.7.1-66-gfc22ab4fd3 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.1-55-g7d5f104fa7 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
OK: 9064 distinct packages available
OK: 23 MiB in 23 packages
Cloning into '/tmp'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
The yaml file which works perfectly for public repo:
apiVersion: batch/v1
kind: Job
metadata:
name: nest-build-kaniko
labels:
app: nest-kaniko-example
spec:
template:
spec:
containers:
-
image: 'gcr.io/kaniko-project/executor:latest'
name: kaniko
args: ["--dockerfile=/workspace/Dockerfile",
"--context=/workspace/",
"--destination=aws.dest.cred"]
volumeMounts:
-
mountPath: /workspace
name: source
-
name: aws-secret
mountPath: /root/.aws/
-
name: docker-config
mountPath: /kaniko/.docker/
initContainers:
-
name: download
image: alpine:3.7
command: ["/bin/sh","-c"]
args: ['apk add --no-cache git && git clone https://github.com/username/repo.git /tmp/']
volumeMounts:
-
mountPath: /tmp
name: source
restartPolicy: Never
volumes:
-
emptyDir: {}
name: source
-
name: aws-secret
secret:
secretName: aws-secret
-
name: docker-config
configMap:
name: docker-config
The yaml file after using git-sync for cloning private repository:
apiVersion: batch/v1
kind: Job
metadata:
name: nest-build-kaniko
labels:
app: nest-kaniko-example
spec:
template:
spec:
containers:
-
image: 'gcr.io/kaniko-project/executor:latest'
name: kaniko
args: ["--dockerfile=/workspace/Dockerfile",
"--context=/workspace/",
"--destination=aws.dest.cred"]
volumeMounts:
-
mountPath: /workspace
name: source
-
name: aws-secret
mountPath: /root/.aws/
-
name: docker-config
mountPath: /kaniko/.docker/
initContainers:
-
name: git-sync
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
volumeMounts:
-
mountPath: /git/tmp
name: source
-
name: git-secret
mountPath: "/etc/git-secret"
env:
- name: GIT_SYNC_REPO
value: "git#gitlab.com:username/repo.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_DEST
value: "/tmp"
- name: GIT_SYNC_ONE_TIME
value: "true"
securityContext:
runAsUser: 0
restartPolicy: Never
volumes:
-
emptyDir: {}
name: source
-
name: aws-secret
secret:
secretName: aws-secret
-
name: git-secret
secret:
secretName: git-creds
defaultMode: 256
-
name: docker-config
configMap:
name: docker-config
You can use git-sync
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: git-sync-test
spec:
selector:
matchLabels:
app: git-sync-test
serviceName: "git-sync-test"
replicas: 1
template:
metadata:
labels:
app: git-sync-test
spec:
containers:
- name: git-sync-test
image: <your-main-image>
volumeMounts:
- name: service
mountPath: /var/magic
initContainers:
- name: git-sync
image: k8s.gcr.io/git-sync-amd64:v2.0.6
imagePullPolicy: Always
volumeMounts:
- name: service
mountPath: /magic
- name: git-secret
mountPath: /etc/git-secret
env:
- name: GIT_SYNC_REPO
value: <repo-path-you-want-to-clone>
- name: GIT_SYNC_BRANCH
value: <repo-branch>
- name: GIT_SYNC_ROOT
value: /magic
- name: GIT_SYNC_DEST
value: <path-where-you-want-to-clone>
- name: GIT_SYNC_PERMISSIONS
value: "0777"
- name: GIT_SYNC_ONE_TIME
value: "true"
- name: GIT_SYNC_SSH
value: "true"
securityContext:
runAsUser: 0
volumes:
- name: service
emptyDir: {}
- name: git-secret
secret:
defaultMode: 256
secretName: git-creds # your-ssh-key
For more details check this link.
initContainers:
-
name: git-sync
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
volumeMounts:
-
mountPath: /workspace
name: source
-
name: git-secret
mountPath: "/etc/git-secret"
env:
- name: GIT_SYNC_REPO
value: "git#gitlab.com:username/repo.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_ROOT
value: /workspace
- name: GIT_SYNC_DEST
value: "tmp"
- name: GIT_SYNC_ONE_TIME
value: "true"
NOTE: set GIT_SYNC_ROOT env to /workspace
It'll clone in /workspace/tmp directory in your emptyDir source volume.
I am running kubernetes 1.9.4 on my gke cluster
I have two pods , gate which is trying to connect to coolapp, both written in elixir
I am using libcluster to connect my nodes
I get the following error:
[libcluster:app_name] cannot query kubernetes (unauthorized): endpoints is forbidden: User "system:serviceaccount:staging:default" cannot list endpoints in the namespace "staging": Unknown user "system:serviceaccount:staging:default"
here is my config in gate under config/prod:
config :libcluster,
topologies: [
app_name: [
strategy: Cluster.Strategy.Kubernetes,
config: [
kubernetes_selector: "tier=backend",
kubernetes_node_basename: System.get_env("MY_POD_NAMESPACE") || "${MY_POD_NAMESPACE}"]]]
here is my configuration:
vm-args
## Name of the node
-name ${MY_POD_NAMESPACE}#${MY_POD_IP}
## Cookie for distributed erlang
-setcookie ${ERLANG_COOKIE}
# Enable SMP automatically based on availability
-smp auto
creating the secrets:
kubectl create secret generic erlang-config --namespace staging --from-literal=erlang-cookie=xxxxxx
kubectl create configmap vm-config --namespace staging --from-file=vm.args
gate/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gate
namespace: staging
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: gate
tier: backend
spec:
securityContext:
runAsUser: 0
runAsNonRoot: false
containers:
- name: gate
image: gcr.io/development/gate:0.1.7
args:
- foreground
ports:
- containerPort: 80
volumeMounts:
- name: config-volume
mountPath: /beamconfig
env:
- name: MY_POD_NAMESPACE
value: staging
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: RELEASE_CONFIG_DIR
value: /beamconfig
- name: ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: erlang-config
key: erlang-cookie
volumes:
- name: config-volume
configMap:
name: vm-config
coolapp/deployment.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: coolapp
namespace: staging
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: coolapp
tier: backend
spec:
securityContext:
runAsUser: 0
runAsNonRoot: false
# volumes
volumes:
- name: config-volume
configMap:
name: vm-config
containers:
- name: coolapp
image: gcr.io/development/coolapp:1.0.3
volumeMounts:
- name: secrets-volume
mountPath: /secrets
readOnly: true
- name: config-volume
mountPath: /beamconfig
ports:
- containerPort: 80
args:
- "foreground"
env:
- name: MY_POD_NAMESPACE
value: staging
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: REPLACE_OS_VARS
value: "true"
- name: RELEASE_CONFIG_DIR
value: /beamconfig
- name: ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: erlang-config
key: erlang-cookie
# proxy_container
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy", "--dir=/cloudsql",
"-instances=staging:us-central1:com-staging=tcp:5432",
"-credential_file=/secrets/cloudsql/credentials.json"]
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
- name: cloudsql
mountPath: /cloudsql
The default service account for the staging namespace (in which apparently your Pods using libcluster are running) lacks RBAC permissions to get endpoints in that namespace.
Likely your application requires a number of other permissions (that are not mentioned in the above error message) to work correctly; identifying all such permissions is out of scope for SO.
A way to resolve this issue is to grant superuser permissions that service account. This is not a secure solution but a stop gap fix.
$ kubectl create clusterrolebinding make-staging-sa-cluster-admin \
--serviceaccount=staging:default \
--clusterrole=cluster-admin
clusterrolebinding "make-staging-sa-cluster-admin" created
To grant the specific permission only (get endpoints in the staging namespace) you would need to create a Role first:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: some-permissions
namespace: staging
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch"]
And create a RoleBinding for the default service account in the staging namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: give-default-sa-some-permissions
namespace: staging
subjects:
- kind: ServiceAccount
name: default
namespace: staging
roleRef:
kind: Role
name: some-permissions
apiGroup: rbac.authorization.k8s.io
Not an erlang/elixir or libcluster user, but it seems it is trying to use the default service account for the namespace to try and query the master for a list of endpoints available in the cluster.
The readme for libcluster says as much:
If set to Cluster.Strategy.Kubernetes, it will use the Kubernetes API
to query endpoints based on a basename and label selector, using the
token and namespace injected into every pod; once it has a list of
endpoints, it uses that list to form a cluster, and keep it up to
date.
Reading the code for the kubernetes.ex in libcluster and the error you get confirm as much.
You will need to setup a ClusterRole and RoleBinding for the service account in the staging namespace. This will allow libcluster to dynamically query the master to discover other erlang nodes in the cluster/namespace.
Here are some handy resources for follow up reading:
https://kubernetes.io/docs/admin/service-accounts-admin/
https://kubernetes.io/docs/admin/authorization/rbac/
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
The reccomended method for managing environment configuration containers running in a pod is through the use of configmap. See the docs here.
This is great although we have containers that require massive amounts of environment variables, this will only expand in the future. Using the prescribed configmap method this become unweildy and hard to manage.
For example a simple deplyment file becomes massive:
apiVersion: v1
kind: Service
metadata:
name: my-app-api
labels:
name: my-app-api
environment: staging
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
name: my-app-api
environment: staging
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-app-api
spec:
replicas: 2
revisionHistoryLimit: 10
template:
metadata:
labels:
name: my-app-api
environment: staging
spec:
containers:
- name: my-app-api
imagePullPolicy: Always
image: myapp/my-app-api:latest
ports:
- containerPort: 80
env:
- name: API_HOST
value: XXXXXXXXXXX
- name: API_ENV
value: XXXXXXXXXXX
- name: API_DEBUG
value: XXXXXXXXXXX
- name: API_KEY
value: XXXXXXXXXXX
- name: EJ_API_ENDPOINT
value: XXXXXXXXXXX
- name: WEB_HOST
value: XXXXXXXXXXX
- name: AWS_ACCESS_KEY
value: XXXXXXXXXXX
- name: AWS_SECRET_KEY
value: XXXXXXXXXXX
- name: CDN
value: XXXXXXXXXXX
- name: STRIPE_KEY
value: XXXXXXXXXXX
- name: STRIPE_SECRET
value: XXXXXXXXXXX
- name: DB_HOST
value: XXXXXXXXXXX
- name: MYSQL_ROOT_PASSWORD
value: XXXXXXXXXXX
- name: MYSQL_DATABASE
value: XXXXXXXXXXX
- name: REDIS_HOST
value: XXXXXXXXXXX
imagePullSecrets:
- name: my-registry-key
Are there alternate easy to inject a central environment configuration?
UPDATE
This was proposed for 1.5 although did not make the cut and looks like it will be included in 1.6. Fingers crossed...
There is a proposal currently targeted for 1.5 that aims to make this easier. As proposed, you would be able to pull all variables from a ConfigMap in one go, without having to spell out each one separately.
If implemented, it would allow you to do something like this:
Warning: This doesn't actually work yet!
ConfigMap:
apiVersion: v1
data:
space-ships: 1
ship-type: battle-cruiser
weapon: laser-cannon
kind: ConfigMap
metadata:
name: space-config
Deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: space-simulator
spec:
template:
metadata:
labels:
app: space-simulator
spec:
containers:
- name: space-simulator
image: foo/space-simulator
# This is the magic piece that would allow you to avoid all that boilerplate!
- envFrom:
configMap: space-config