Openshift oc patch not executing initdb.sql from /docker-entrypoint-initdb.d - docker

OpenShift:
I have the below MySQL Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-master
spec:
selector:
matchLabels:
app: mysql-master
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql-master
spec:
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: ro-mstr-nfs-datadir-claim
containers:
- image: mysql:5.7
name: mysql-master
env:
- name: MYSQL_SERVER_CONTAINER
value: mysql
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_ROOT_PASSWORD
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_DATABASE
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_USER
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_PASSWORD
ports:
- containerPort: 3306
name: mysql-master
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
I created a deployment using this yml file which created a deployment and pod which is successfully running.
And I have a configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: ro-mstr-mysqlinitcnfgmap
data:
initdb.sql: |-
CREATE TABLE aadhaar ( name varchar(255) NOT NULL,
sex char NOT NULL, birth DATE NOT NULL, death DATE NULL,
id int(255) NOT NULL AUTO_INCREMENT, PRIMARY KEY (id) );
CREATE USER 'usera'#'%' IDENTIFIED BY 'usera';
GRANT REPLICATION SLAVE ON *.* TO 'usera' IDENTIFIED BY 'usera';
FLUSH PRIVILEGES;
Now I need to patch the above deployment using this configmap. I am using the below command
oc patch deployment mysql-master -p '{ "spec": { "template": { "spec": { "volumes": [ { "name": "ro-mysqlinitconf-vol", "configMap": { "name": "ro-mstr-mysqlinitcnfgmap" } } ], "containers": [ { "image": "mysql:5.7", "name": "mysql-master", "volumeMounts": [ { "name": "ro-mysqlinitconf-vol", "mountPath": "/docker-entrypoint-initdb.d" } ] } ] } } } }'
So the above command is successful, I validated the Deployment description and inside the container it placed the initdb.sql file successfully, and recreated the pod. But the issue is it has not created the aadhaar table. I think it has not executed the initdb.sql file from docker-entrypoint-initdb.d.

If you dive into the entrypoint script in your image (https://github.com/docker-library/mysql/blob/75f81c8e20e5085422155c48a50d99321212bf6f/5.7/docker-entrypoint.sh#L341-L350) you can see it only runs the initdb.d files if it is also creating the database the first time. I think maybe you assumed it always ran them on startup?

Related

VerneMQ not kicking client with revoked certificate

I have setup VerneMQ to pull CRL (Certificate Revocation List) from Vault. Followingis my setup manifest.
kind: ConfigMap
apiVersion: v1
metadata:
name: vernemq-refresh-crl
namespace: backend
labels:
app: vernemq
data:
pull_crl.sh: |
#!/usr/bin/env sh
if ! apk info | grep ^curl ; then apk update && apk add curl; fi
while true
do
echo $(date)
curl -w "\n" --header "X-Vault-Token: $VAULT_TOKEN" http://vault.backend.svc.cluster.local:8200/v1/my-ca/crl/pem > /tmp/shared/ca.crl
sleep $SLEEP_INTERVAL
done
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vernemq
namespace: backend
spec:
replicas: 1
selector:
matchLabels:
app: vernemq
template:
metadata:
labels:
app: vernemq
spec:
serviceAccountName: vernemq
containers:
- name: vernemq
image: vernemq/vernemq:1.12.3
ports:
- name: mqtt
containerPort: 1883
- name: mqtts
containerPort: 8883
- name: mqtt-ws
containerPort: 8080
- name: epmd
containerPort: 4369
- name: vmq
containerPort: 44053
- name: metrics
containerPort: 8888
env:
- name: DOCKER_VERNEMQ_ACCEPT_EULA
value: "yes"
- name: MY_POD_NAME
value: "vernemq"
- name: DOCKER_VERNEMQ_KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: DOCKER_VERNEMQ_KUBERNETES_APP_LABEL
value: "vernemq"
- name: DOCKER_VERNEMQ_LOG__CONSOLE__LEVEL
value: "debug"
- name: DOCKER_VERNEMQ_KUBERNETES_LABEL_SELECTOR
value: "app=vernemq"
- name: DOCKER_VERNEMQ_LISTENER__TCP__ALLOWED_PROTOCOL_VERSIONS
value: "3,4,5"
- name: DOCKER_VERNEMQ_LISTENER__SSL__ALLOWED_PROTOCOL_VERSIONS
value: "3,4,5"
- name: DOCKER_VERNEMQ_ALLOW_ANONYMOUS
value: "on"
- name: DOCKER_VERNEMQ_TOPIC_MAX_DEPTH
value: "20"
- name: DOCKER_VERNEMQ_KUBERNETES_INSECURE
value: "1"
- name: DOCKER_VERNEMQ_MAX_ONLINE_MESSAGES
value: "-1"
- name: DOCKER_VERNEMQ_MAX_OFFLINE_MESSAGES
value: "-1"
- name: DOCKER_VERNEMQ_MAX_INFLIGHT_MESSAGES
value: "0"
- name: DOCKER_VERNEMQ_LISTENER__TCP__DEFAULT
value: "0.0.0.0:1883"
- name: DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT
value: "0.0.0.0:8883"
- name: DOCKER_VERNEMQ_LISTENER__WS__DEFAULT
value: "0.0.0.0:8080"
- name: DOCKER_VERNEMQ_LISTENER__HTTP__METRICS
value: "0.0.0.0:8888"
- name: DOCKER_VERNEMQ_LISTENER__HTTP__DEFAULT
value: "0.0.0.0:8888"
- name: DOCKER_VERNEMQ_LISTENER__SSL__REQUIRE_CERTIFICATE
value: "on"
- name: DOCKER_VERNEMQ_LISTENER__SSL__USE_IDENTITY_AS_USERNAME
value: "on"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CAFILE
value: "/vernemq/cert/ca.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE
value: "/vernemq/cert/server.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE
value: "/vernemq/cert/server.key"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CRLFILE
value: "/tmp/shared/ca.crl"
volumeMounts:
- name: cert
mountPath: /vernemq/cert
readOnly: true
- name: acl
mountPath: /vernemq/acl
readOnly: true
- name: tmp-shared
mountPath: /tmp/shared
- name: pull-crl
image: alpine
command: ["/bin/sh"]
args: ["-c", "/script/pull_crl.sh"]
env:
- name: VAULT_TOKEN
valueFrom:
secretKeyRef:
name: vault
key: root-token
- name: SLEEP_INTERVAL
value: "300"
volumeMounts:
- name: pull-crl-sh
mountPath: /script/pull_crl.sh
subPath: pull_crl.sh
- name: tmp-shared
mountPath: /tmp/shared
volumes:
- name: cert
secret:
secretName: vernemq-cert
- name: pull-crl-sh
configMap:
name: vernemq-refresh-crl
defaultMode: 0744
- name: tmp-shared
emptyDir: {}
I do see serial number of revoked certificate in /tmp/shared/ca.crl but client using the revoked certificate is able to maintain already establish session with broker.
Is there a easy way to kick the client with revoked certificate
Check the administrative disconnect command with sudo vmq-admin session disconnect --help.
In general, it is a wrong assumption that adding a cert to the CRL will disrupt an existing connection. You have to couple that step with an administrative disconnect of the client.

Bitnami airflow scheduler could not connect database while webserve can connect even they have same properties?

I want to configure airflow on openshift.
I set database on openshift like below :
kind: Service
apiVersion: v1
metadata:
name: airflow-database
namespace: ersin-poc
spec:
ports:
- name: 5432-tcp
protocol: TCP
port: 5432
targetPort: 5432
selector:
deployment: airflow-database
type: ClusterIP
sessionAffinity: None
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
and my database deployment like below :
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-database
namespace: ersin-poc
labels:
deployment: airflow-database
spec:
replicas: 1
selector:
matchLabels:
deployment: airflow-database
template:
metadata:
creationTimestamp: null
labels:
deployment: airflow-database
spec:
volumes:
- name: generic
persistentVolumeClaim:
claimName: generic
- name: empty1
emptyDir: {}
containers:
- resources: {}
name: airflow-database
env:
- name: POSTGRESQL_USERNAME
value: 'bn_airflow'
- name: POSTGRESQL_PASSWORD
value: 'bitnami1'
- name: POSTGRESQL_DATABASE
value: 'bitnami_airflow'
ports:
- containerPort: 5432
protocol: TCP
volumeMounts:
- name: generic
mountPath: /bitnami/postgresql/
image: >-
bitnami/postgresql:latest
hostname: airflow-database
I can connect this db from my webserver like below :
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-webserver
namespace: ersin-poc
labels:
deployment: airflow-webserver
spec:
replicas: 1
selector:
matchLabels:
deployment: airflow-webserver
template:
metadata:
creationTimestamp: null
labels:
deployment: airflow-webserver
spec:
volumes:
- name: generic
persistentVolumeClaim:
claimName: generic
- name: empty1
emptyDir: {}
containers:
- resources: {}
name: airflow-webserver
env:
- name: AIRFLOW_HOME
value: /home/appuser
- name: USER
value: appuser
- name: AIRFLOW_FERNET_KEY
value: '46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho='
- name: AIRFLOW_SECRET_KEY
value: 'a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08='
- name: AIRFLOW_EXECUTOR
value: 'CeleryExecutor'
- name: AIRFLOW_DATABASE_NAME
value: 'bitnami_airflow'
- name: AIRFLOW_DATABASE_USERNAME
value: 'bn_airflow'
- name: AIRFLOW_DATABASE_PASSWORD
value: 'bitnami1'
- name: AIRFLOW_LOAD_EXAMPLES
value: 'yes'
- name: AIRFLOW_PASSWORD
value: 'bitnami123'
- name: AIRFLOW_USERNAME
value: 'user'
- name: AIRFLOW_EMAIL
value: 'user#example.com'
- name: AIRFLOW_DATABASE_HOST
value: 'airflow-database'
- name: AIRFLOW_DATABASE_PORT_NUMBER
value: '5432'
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: generic
mountPath: /home/appuser
- name: generic
mountPath: /home/appuser/logs/
- name: generic
mountPath: /home/appuser/dags/
image: >-
bitnami/airflow:latest
hostname: airflow-webserver
but when i try it with airflow-scheduler it gives error :
airflow-scheduler 09:29:43.31 INFO ==> Trying to connect to the database server airflow-scheduler 09:30:47.42 ERROR ==> Could not connect to the database
and my scheduler yaml is :
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-scheduler
namespace: ersin-poc
labels:
deployment: airflow-scheduler
spec:
replicas: 1
selector:
matchLabels:
deployment: airflow-scheduler
template:
metadata:
labels:
deployment: airflow-scheduler
spec:
volumes:
- name: generic
persistentVolumeClaim:
claimName: generic
- name: empty1
emptyDir: {}
containers:
- resources: {}
name: airflow-scheduler
env:
- name: AIRFLOW_HOME
value: /home/appuser
- name: USER
value: appuser
- name: AIRFLOW_FERNET_KEY
value: '46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho='
- name: AIRFLOW_SECRET_KEY
value: 'a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08='
- name: AIRFLOW_EXECUTOR
value: 'CeleryExecutor'
- name: AIRFLOW_DATABASE_NAME
value: 'bitnami_airflow'
- name: AIRFLOW_DATABASE_USERNAME
value: 'bn_airflow'
- name: AIRFLOW_DATABASE_PASSWORD
value: 'bitnami1'
- name: AIRFLOW_DATABASE_HOST
value: 'airflow-database'
- name: AIRFLOW_DATABASE_PORT_NUMBER
value: '5432'
- name: AIRFLOW_WEBSERVER_HOST
value: 'airflow-webserver'
- name: AIRFLOW_WEBSERVER_PORT_NUMBER
value: '8080'
- name: REDIS_HOST
value: 'airflow-redis'
- name: REDIS_PORT_NUMBER
value: '6379'
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: generic
mountPath: /home/appuser
- name: generic
mountPath: /home/appuser/logs/
- name: generic
mountPath: /home/appuser/dags/
image: >-
bitnami/airflow-scheduler:latest
hostname: airflow-scheduler
so i cant understand why i got this error with same properties?
thanks in advance
EDIT
and I try in scheduler pod this commands to see whether i can connect to db or not :
psql -h airflow-database -p 5432 -U bn_airflow -d bitnami_airflow -W
pass: bitnami1
select * from public.ab_user;
and yes I can.
After a lot of search , I ve decided to make this with apache/airflow images. (posgresql and redis are still bitnami - it doesnt important)
You can see all ymal files about airflow on openshift.
https://github.com/ersingulbahar/airflow_on_openshift
It works now as expected

Clarification on the security of using secretKeyRef in Kubernetes manifest

I was looking into an entirely separate issue and then came across this question which raised some concerns:
https://stackoverflow.com/a/50510753/3123109
I'm doing something pretty similar. I'm using the CSI Driver for Azure to integrate Azure Kubernetes Service with Azure Key Vault. My manifests for the integration are something like:
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
name: aks-akv-identity
namespace: prod
spec:
type: 0
resourceID: $identityResourceId
clientID: $identityClientId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
name: aks-akv-identity-binding
namespace: prod
spec:
azureIdentity: aks-akv-identity
selector: aks-akv-identity-binding-selector
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: aks-akv-secret-provider
namespace: prod
spec:
provider: azure
secretObjects:
- secretName: ${resourcePrefix}-prod-secrets
type: Opaque
data:
- objectName: PROD-PGDATABASE
key: PGDATABASE
- objectName: PROD-PGHOST
key: PGHOST
- objectName: PROD-PGPORT
key: PGPORT
- objectName: PROD-PGUSER
key: PGUSER
- objectName: PROD-PGPASSWORD
key: PGPASSWORD
parameters:
usePodIdentity: "true"
keyvaultName: ${resourceGroupName}akv
cloudName: ""
objects: |
array:
objectName: PROD-PGDATABASE
objectType: secret
objectVersion: ""
- |
objectName: PROD-PGHOST
objectType: secret
objectVersion: ""
- |
objectName: PROD-PGPORT
objectType: secret
objectVersion: ""
- |
objectName: PROD-PGUSER
objectType: secret
objectVersion: ""
- |
objectName: PROD-PGPASSWORD
objectType: secret
objectVersion: ""
tenantId: $tenantId
Then in the micro service manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-deployment-prod
namespace: prod
spec:
replicas: 3
selector:
matchLabels:
component: api
template:
metadata:
labels:
component: api
aadpodidbinding: aks-akv-identity-binding-selector
spec:
containers:
- name: api
image: appacr.azurecr.io/app-api
ports:
- containerPort: 5000
env:
- name: PGDATABASE
valueFrom:
secretKeyRef:
name: app-prod-secrets
key: PGDATABASE
- name: PGHOST
value: postgres-cluster-ip-service-prod
- name: PGPORT
valueFrom:
secretKeyRef:
name: app-prod-secrets
key: PGPORT
- name: PGUSER
valueFrom:
secretKeyRef:
name: app-prod-secrets
key: PGUSER
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: app-prod-secrets
key: PGPASSWORD
volumeMounts:
- name: secrets-store01-inline
mountPath: /mnt/secrets-store
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aks-akv-secret-provider
---
apiVersion: v1
kind: Service
metadata:
name: api-cluster-ip-service-prod
namespace: prod
spec:
type: ClusterIP
selector:
component: api
ports:
- port: 5000
targetPort: 5000
Then in my application settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ['PGDATABASE'],
'USER': os.environ['PGUSER'],
'PASSWORD': os.environ['PGPASSWORD'],
'HOST': os.environ['PGHOST'],
'PORT': os.environ['PGPORT'],
}
}
Nothing in my Dockerfile refers to any of these variables, just the Django micro service code.
According to the link, one of the comments was:
current best practices advise against doing this exactly. secrets managed through environment variables in docker are easily viewed and should not be considered secure.
So I'm second guessing this approach.
Do I need to look into revising what I have here?
The suggestion in the link is to place the os.environ[] with a call to a method that pulls the credentials from a key vault... but the credentials to even access the key vault would need to be stored in secrets... so I'm not seeing how it is any different.
Note: One thing I noticed is this is the use of env: and mounting the secrets to a volume is redundant. The latter was done per the documentation on the integration, but it makes the secrets available from /mnt/secrets-store so you can do something like cat /mnt/secrets-store/PROD-PGUSER. os.environ[] isn't really necessary and the env: I don't think because you could pull the secret from that location in the Pod.
At least doing something like the following prints out the secret value:
kubectl exec -it $(kubectl get pods -l component=api -o custom-columns=:metadata.name -n prod) -n prod -- cat /mnt/secrets-store/PROD-PGUSER
The comment on the answer you linked was incorrect. I've left a note to explain the confusion. What you have is fine, if possibly over-built :) You're not actually gaining any security vs. just using Kubernetes Secrets directly but if you prefer the workflow around AKV then this looks fine. You might want to look at externalsecrets rather than this weird side feature of the CSI stuff? The CSI driver is more for exposing stuff as files rather than external->Secret->envvar.

How to clone a private git repository into a kubernetes pod using ssh keys in secrets?

I am trying to clone a private git repository(gitLab) into a kubernetes pod, using SSH keys for authentication. I have stored my keys in a secret. Here is the yaml file for the job that does the desired task.
Heres the same question, but doesnt give the exact solution :
Clone a secure git repo in Kubernetes pod
Logs of the init container after execution:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
v3.7.1-66-gfc22ab4fd3 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.1-55-g7d5f104fa7 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
OK: 9064 distinct packages available
OK: 23 MiB in 23 packages
Cloning into '/tmp'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
The yaml file which works perfectly for public repo:
apiVersion: batch/v1
kind: Job
metadata:
name: nest-build-kaniko
labels:
app: nest-kaniko-example
spec:
template:
spec:
containers:
-
image: 'gcr.io/kaniko-project/executor:latest'
name: kaniko
args: ["--dockerfile=/workspace/Dockerfile",
"--context=/workspace/",
"--destination=aws.dest.cred"]
volumeMounts:
-
mountPath: /workspace
name: source
-
name: aws-secret
mountPath: /root/.aws/
-
name: docker-config
mountPath: /kaniko/.docker/
initContainers:
-
name: download
image: alpine:3.7
command: ["/bin/sh","-c"]
args: ['apk add --no-cache git && git clone https://github.com/username/repo.git /tmp/']
volumeMounts:
-
mountPath: /tmp
name: source
restartPolicy: Never
volumes:
-
emptyDir: {}
name: source
-
name: aws-secret
secret:
secretName: aws-secret
-
name: docker-config
configMap:
name: docker-config
The yaml file after using git-sync for cloning private repository:
apiVersion: batch/v1
kind: Job
metadata:
name: nest-build-kaniko
labels:
app: nest-kaniko-example
spec:
template:
spec:
containers:
-
image: 'gcr.io/kaniko-project/executor:latest'
name: kaniko
args: ["--dockerfile=/workspace/Dockerfile",
"--context=/workspace/",
"--destination=aws.dest.cred"]
volumeMounts:
-
mountPath: /workspace
name: source
-
name: aws-secret
mountPath: /root/.aws/
-
name: docker-config
mountPath: /kaniko/.docker/
initContainers:
-
name: git-sync
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
volumeMounts:
-
mountPath: /git/tmp
name: source
-
name: git-secret
mountPath: "/etc/git-secret"
env:
- name: GIT_SYNC_REPO
value: "git#gitlab.com:username/repo.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_DEST
value: "/tmp"
- name: GIT_SYNC_ONE_TIME
value: "true"
securityContext:
runAsUser: 0
restartPolicy: Never
volumes:
-
emptyDir: {}
name: source
-
name: aws-secret
secret:
secretName: aws-secret
-
name: git-secret
secret:
secretName: git-creds
defaultMode: 256
-
name: docker-config
configMap:
name: docker-config
You can use git-sync
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: git-sync-test
spec:
selector:
matchLabels:
app: git-sync-test
serviceName: "git-sync-test"
replicas: 1
template:
metadata:
labels:
app: git-sync-test
spec:
containers:
- name: git-sync-test
image: <your-main-image>
volumeMounts:
- name: service
mountPath: /var/magic
initContainers:
- name: git-sync
image: k8s.gcr.io/git-sync-amd64:v2.0.6
imagePullPolicy: Always
volumeMounts:
- name: service
mountPath: /magic
- name: git-secret
mountPath: /etc/git-secret
env:
- name: GIT_SYNC_REPO
value: <repo-path-you-want-to-clone>
- name: GIT_SYNC_BRANCH
value: <repo-branch>
- name: GIT_SYNC_ROOT
value: /magic
- name: GIT_SYNC_DEST
value: <path-where-you-want-to-clone>
- name: GIT_SYNC_PERMISSIONS
value: "0777"
- name: GIT_SYNC_ONE_TIME
value: "true"
- name: GIT_SYNC_SSH
value: "true"
securityContext:
runAsUser: 0
volumes:
- name: service
emptyDir: {}
- name: git-secret
secret:
defaultMode: 256
secretName: git-creds # your-ssh-key
For more details check this link.
initContainers:
-
name: git-sync
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
volumeMounts:
-
mountPath: /workspace
name: source
-
name: git-secret
mountPath: "/etc/git-secret"
env:
- name: GIT_SYNC_REPO
value: "git#gitlab.com:username/repo.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_ROOT
value: /workspace
- name: GIT_SYNC_DEST
value: "tmp"
- name: GIT_SYNC_ONE_TIME
value: "true"
NOTE: set GIT_SYNC_ROOT env to /workspace
It'll clone in /workspace/tmp directory in your emptyDir source volume.

cannot query kubernetes (unauthorized): endpoints is forbidden: User cannot list endpoints in the namespace

I am running kubernetes 1.9.4 on my gke cluster
I have two pods , gate which is trying to connect to coolapp, both written in elixir
I am using libcluster to connect my nodes
I get the following error:
[libcluster:app_name] cannot query kubernetes (unauthorized): endpoints is forbidden: User "system:serviceaccount:staging:default" cannot list endpoints in the namespace "staging": Unknown user "system:serviceaccount:staging:default"
here is my config in gate under config/prod:
config :libcluster,
topologies: [
app_name: [
strategy: Cluster.Strategy.Kubernetes,
config: [
kubernetes_selector: "tier=backend",
kubernetes_node_basename: System.get_env("MY_POD_NAMESPACE") || "${MY_POD_NAMESPACE}"]]]
here is my configuration:
vm-args
## Name of the node
-name ${MY_POD_NAMESPACE}#${MY_POD_IP}
## Cookie for distributed erlang
-setcookie ${ERLANG_COOKIE}
# Enable SMP automatically based on availability
-smp auto
creating the secrets:
kubectl create secret generic erlang-config --namespace staging --from-literal=erlang-cookie=xxxxxx
kubectl create configmap vm-config --namespace staging --from-file=vm.args
gate/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gate
namespace: staging
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: gate
tier: backend
spec:
securityContext:
runAsUser: 0
runAsNonRoot: false
containers:
- name: gate
image: gcr.io/development/gate:0.1.7
args:
- foreground
ports:
- containerPort: 80
volumeMounts:
- name: config-volume
mountPath: /beamconfig
env:
- name: MY_POD_NAMESPACE
value: staging
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: RELEASE_CONFIG_DIR
value: /beamconfig
- name: ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: erlang-config
key: erlang-cookie
volumes:
- name: config-volume
configMap:
name: vm-config
coolapp/deployment.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: coolapp
namespace: staging
spec:
replicas: 1
revisionHistoryLimit: 1
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: coolapp
tier: backend
spec:
securityContext:
runAsUser: 0
runAsNonRoot: false
# volumes
volumes:
- name: config-volume
configMap:
name: vm-config
containers:
- name: coolapp
image: gcr.io/development/coolapp:1.0.3
volumeMounts:
- name: secrets-volume
mountPath: /secrets
readOnly: true
- name: config-volume
mountPath: /beamconfig
ports:
- containerPort: 80
args:
- "foreground"
env:
- name: MY_POD_NAMESPACE
value: staging
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: REPLACE_OS_VARS
value: "true"
- name: RELEASE_CONFIG_DIR
value: /beamconfig
- name: ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: erlang-config
key: erlang-cookie
# proxy_container
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy", "--dir=/cloudsql",
"-instances=staging:us-central1:com-staging=tcp:5432",
"-credential_file=/secrets/cloudsql/credentials.json"]
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
- name: cloudsql
mountPath: /cloudsql
The default service account for the staging namespace (in which apparently your Pods using libcluster are running) lacks RBAC permissions to get endpoints in that namespace.
Likely your application requires a number of other permissions (that are not mentioned in the above error message) to work correctly; identifying all such permissions is out of scope for SO.
A way to resolve this issue is to grant superuser permissions that service account. This is not a secure solution but a stop gap fix.
$ kubectl create clusterrolebinding make-staging-sa-cluster-admin \
--serviceaccount=staging:default \
--clusterrole=cluster-admin
clusterrolebinding "make-staging-sa-cluster-admin" created
To grant the specific permission only (get endpoints in the staging namespace) you would need to create a Role first:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: some-permissions
namespace: staging
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch"]
And create a RoleBinding for the default service account in the staging namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: give-default-sa-some-permissions
namespace: staging
subjects:
- kind: ServiceAccount
name: default
namespace: staging
roleRef:
kind: Role
name: some-permissions
apiGroup: rbac.authorization.k8s.io
Not an erlang/elixir or libcluster user, but it seems it is trying to use the default service account for the namespace to try and query the master for a list of endpoints available in the cluster.
The readme for libcluster says as much:
If set to Cluster.Strategy.Kubernetes, it will use the Kubernetes API
to query endpoints based on a basename and label selector, using the
token and namespace injected into every pod; once it has a list of
endpoints, it uses that list to form a cluster, and keep it up to
date.
Reading the code for the kubernetes.ex in libcluster and the error you get confirm as much.
You will need to setup a ClusterRole and RoleBinding for the service account in the staging namespace. This will allow libcluster to dynamically query the master to discover other erlang nodes in the cluster/namespace.
Here are some handy resources for follow up reading:
https://kubernetes.io/docs/admin/service-accounts-admin/
https://kubernetes.io/docs/admin/authorization/rbac/
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

Resources