Problem with definition of Kubernetes Ingress in helm - docker

I'm trying to deploy the following Ingress with helm:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
io.ctl.cd/ssl: "ui.releasename"
name: ui
labels:
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
spec:
rules:
{{ if eq .Values.nodeSelector.location "minikube" }}
- host: ui.{{ .Release.Namespace }}.minikube.test
{{ else }}
- host: ui.{{ .Release.Namespace }}.devhost
{{ end }}
http:
paths:
- backend:
serviceName: api
servicePort: {{ .Values.api.service.port }}
path: /
And I'm getting the following error
Error: release x-**** failed: Ingress in version "v1beta1" cannot be handled as a Ingress: only encoded map or array can be decoded into a struct
I have a very similar ingress that is working fine, I don't don't want is happening with this one.

I think problem in this string:
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
For test, try:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
io.ctl.cd/ssl: "ui.releasename"
name: ui
labels:
chart: "{{ .Chart.Name }}"
spec:
rules:
{{ if eq .Values.nodeSelector.location "minikube" }}
- host: ui.{{ .Release.Namespace }}.minikube.test
{{ else }}
- host: ui.{{ .Release.Namespace }}.devhost
{{ end }}
http:
paths:
- backend:
serviceName: api
servicePort: {{ .Values.api.service.port }}
path: /

Related

Unable to pull from Gitlab Container Registry unless set to Everyone With Access

I am working on building a simple pipeline with Gitlab. I'm using Minikube on my laptop and I've installed gitlab-runner using helm on the same namespace of the application I'm trying to deploy. I've not installed Gitlab on Minikube, I'm using Gitlab.com.
Anyway, after a lot of attempts, the deployment was successful and the application was deployed but failed because it can't pull the image from the registry.gitlab.com. The error is repository does not exist or may require 'docker login': denied: requested access to the resource is denied
I've also logged in successfully with docker login registry.gitlab.com -u username -p pwd but I can't pull the image, same error as above.
I've created secrets according to the documentation. Here's my deployment file
apiVersion: v1
kind: Secret
metadata:
name: registry-credentials
namespace: {{ .Values.applicationName }}
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: ..hidden..
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.applicationName }}
namespace: {{ .Values.applicationName }}
spec:
replicas: 1
selector:
matchLabels:
app: {{ .Values.applicationName }}
template:
metadata:
labels:
app: {{ .Values.applicationName }}
spec:
containers:
- name: {{ .Values.applicationName }}
image: registry.gitlab.com/gfalco77/maverick:latest
imagePullPolicy: Always
ports:
- containerPort: 8001
imagePullSecrets:
- name: registry-credentials
---
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.applicationName }}
spec:
ports:
- name: {{ .Values.applicationName }}
port: 8001
targetPort: 8001
protocol: TCP
selector:
app: {{ .Values.applicationName }}
I've also created the deploy token with read_registry.
Project visibility is already Public but container registry was set to 'Only Project Members'
Only way I can make it work is to change the permissions of the container registry to Everyone With Access.
Is this obvious or it can also be done with permissions 'Only project members'?
Thanks

How do I add a foreground process to a Docker container

I am facing a "CrashLoopBackoff" error when I deploy a .Net Core API with helm upgrade --install flextoeco . :
NAME READY STATUS RESTARTS AGE
flextoecoapi-6bb7cdd846-r6c67 0/1 CrashLoopBackOff 4 (38s ago) 3m8s
flextoecoapi-fb7f7b556-tgbrv 0/1 CrashLoopBackOff 219 (53s ago) 10h
mssql-depl-86c86b5f44-ldj48 0/1 Pending
I have run ks describe pod flextoecoapi-6bb7cdd846-r6c67 and part of the output is as below :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m4s default-scheduler Successfully assigned default/flextoecoapi-6bb7cdd846-r6c67 to fbcdcesdn02
Normal Pulling 5m3s kubelet Pulling image "golide/flextoeco:1.1.1"
Normal Pulled 4m57s kubelet Successfully pulled image "golide/flextoeco:1.1.1" in 6.2802081s
Normal Killing 4m34s kubelet Container flextoeco failed liveness probe, will be restarted
Normal Created 4m33s (x2 over 4m57s) kubelet Created container flextoeco
Normal Started 4m33s (x2 over 4m56s) kubelet Started container flextoeco
Normal Pulled 4m33s kubelet Container image "golide/flextoeco:1.1.1" already present on machine
Warning Unhealthy 4m14s (x12 over 4m56s) kubelet Readiness probe failed: Get "http://10.244.6.59:80/": dial tcp 10.244.0.59:80: connect: connection refused
Warning Unhealthy 4m14s (x5 over 4m54s) kubelet Liveness probe failed: Get "http://10.244.6.59:80/": dial tcp 10.244.0.59:80: connect: connection refused
Warning BackOff 3s (x10 over 2m33s) kubelet Back-off restarting failed container
Taking from the suggestions here it appears I have a number of options to fix most notable being:
i) Add a command to the Dockerfile that will ensure there is some foreground process running
ii)Extend the LivenessProbe initialDelaySeconds
I have opted for the first and edited my Dockerfile as below :
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:3.1
WORKDIR /app
ENV ASPNETCORE_URLS http://+:5000
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "FlexToEcocash.dll"]
CMD tail -f /dev/null
After this change I am still getting the same error.
UPDATE
Skipped : The deployment works perfectly when I am not using helm i.e I can do a kubectl apply for the deployment/service/nodeport/clusterip and the API is deployed without issues.
I have tried to update values.yaml and service.yaml as below, but after redeploy the CrashLoopBackOff error persists :
templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ include "flextoeco.fullname" . }}
labels:
{{- include "flextoeco.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "flextoeco.selectorLabels" . | nindent 4 }}
values.yaml
I have explicitly specified the CPU and memory usage here
replicaCount: 1
image:
repository: golide/flextoeco
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: "1.1.2"
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
service:
type: ClusterIP
port: 80
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: flextoeco.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 100m
memory: 250Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
nodeSelector: {}
templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "flextoeco.fullname" . }}
labels:
{{- include "flextoeco.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "flextoeco.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "flextoeco.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "flextoeco.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
labels:
{{- include "flextoeco.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "flextoeco.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 80
protocol: TCP
livenessProbe:
tcpSocket:
port: 8085
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 20
readinessProbe:
tcpSocket:
port: 8085
initialDelaySeconds: 300
periodSeconds: 30
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
{{- toYaml . | nindent 8 }}
{{- end }}
In the Deployment spec, I need to use port 5000 as the containerPort: value and also the port: in the probes. My application is listening on port 5000 :
- name: http
containerPort: 5000
protocol: TCP
livenessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 20
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 300
periodSeconds: 30
The configuration in service.yaml is correct : If the Deployment spec maps the name http to port 5000 then referring to targetPort: http in the Service is right.

Run docker commands within pod using helm

I am trying to run a docker container within a job that I am deploying with helm using AKS. The purpose of this is to run some tests using Selenium and make some postgres calls to automate web ui tests.
When trying to run within the job, the following error is received:
"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
I ran into this problem locally, but can work around it using
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock web-ui-auto:latest /bin/bash
The problem is I am using helm to deploy the job separate from the running tasks since it can take about an hour to complete.
I tried adding a deployment.yaml to my helm like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "web-ui-auto.fullname" . }}
labels:
app: {{ template "web-ui-auto.name" . }}
chart: {{ template "web-ui-auto.chart" . }}
draft: {{ .Values.draft | default "draft-app" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
revisionHistoryLimit: 5
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "web-ui-auto.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "web-ui-auto.name" . }}
draft: {{ .Values.draft | default "draft-app" }}
release: {{ .Release.Name }}
annotations:
buildID: {{ .Values.buildID | default "" | quote }}
container.apparmor.security.beta.kubernetes.io/{{ .Chart.Name }}: runtime/default
spec:
containers:
- name: {{ .Chart.Name }}
securityContext:
runAsNonRoot: {{ .Values.securityContext.runAsNonRoot }}
runAsUser: {{ .Values.securityContext.runAsUser }}
runAsGroup: {{ .Values.securityContext.runAsGroup }}
allowPrivilegeEscalation: {{ .Values.securityContext.allowPrivilegeEscalation }}
seccompProfile:
type: RuntimeDefault
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.deployment.containerPort }}
protocol: TCP
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
but was met with failures still. My question is what is the best method to run docker within the job successfully when deploying with helm. Any help is appreciated.

Converting docker-compose to a helm chart?

I have a docker-compose file containing 2 images to a security tool I am using. My challenge is to convert it into helm chart consisting of deployment.yaml and service.yaml. The docker-compose looks like this -
version: '3'
services:
nginx:
ports:
- "80:80"
- "443:443"
environment:
- NG_SERVER_NAME=192.168.1.228
links:
- tomcat8
image: continuumsecurity/iriusrisk-prod:nginx-prod-ssl
container_name: iriusrisk-nginx
volumes:
- "./cert.pem:/etc/nginx/ssl/star_iriusrisk_com.crt"
- "./key.pem:/etc/nginx/ssl/star_iriusrisk_com.key"
tomcat8:
environment:
- IRIUS_DB_URL=jdbc\:postgresql\://192.168.1.228\:5432/iriusprod?user\=iriusprod&password\=alongandcomplexpassword2523
- IRIUS_EDITION=saas
- IRIUS_EXT_URL=http\://192.168.1.228
- grails_env=production
image: continuumsecurity/iriusrisk-prod:tomcat8-2
container_name: iriusrisk-tomcat8
There is a postgres server running too which I am able to convert into a helm chart and expose it to my ip (192.168.1.228) on port 5432. But for the iriusrisk and tomcat image which are linked to each other, I am not able to it figure out. This has been my solution for the deployment file for both.
deployment-tomcat.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat
labels:
app: {{ .Values.tomcat.app.name }}
spec:
replicas: {{ .Values.tomcat.replicas }}
selector:
matchLabels:
app: {{ .Values.tomcat.app.name }}
template:
metadata:
labels:
app: {{ .Values.tomcat.app.name }}
spec:
{{- if .Values.tomcat.imagePullSecretsName }}
imagePullSecrets:
- name: {{ .Values.tomcat.imagePullSecretsName }}
{{- end}}
restartPolicy: Always
serviceAccountName: {{ .Values.tomcat.serviceAccountName }}
containers:
- name: {{ .Values.tomcat.app.name }}
image: "{{ .Values.tomcat.ImageName }}:{{ .Values.tomcat.ImageTag }}"
container_name: iriusrisk-tomcat8
imagePullPolicy: {{ .Values.tomcat.ImagePullPolicy }}
ports:
- containerPort: {{ .Values.tomcat.port }}
env:
- name: IRIUS_DB_URL
value: jdbc\:postgresql\://192.168.1.228\:5432/iriusprod?user\=iriusprod&password\=alongandcomplexpassword2523
- name: IRIUS_EDITION
value: saas
- name: IRIUS_EXT_URL
value: http\://192.168.1.228
- name: grails_env
value: production
deployment-iriusrisk.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iriusrisk
labels:
app: {{ .Values.iriusrisk.app.name }}
spec:
replicas: {{ .Values.iriusrisk.replicas }}
selector:
matchLabels:
app: {{ .Values.iriusrisk.app.name }}
template:
metadata:
labels:
app: {{ .Values.iriusrisk.app.name }}
spec:
{{- if .Values.iriusrisk.imagePullSecretsName }}
imagePullSecrets:
- name: {{ .Values.iriusrisk.imagePullSecretsName }}
{{- end}}
restartPolicy: Always
serviceAccountName: {{ .Values.iriusrisk.serviceAccountName }}
containers:
- name: {{ .Values.iriusrisk.app.name }}
image: "{{ .Values.iriusrisk.ImageName }}:{{ .Values.iriusrisk.ImageTag }}"
container_name: iriusrisk-nginx
imagePullPolicy: {{ .Values.iriusrisk.ImagePullPolicy }}
ports:
- containerPort: {{ .Values.iriusrisk.port }}
env:
- name: NG_SERVER_NAME
value: "192.168.1.228"
volumes:
- "./cert.pem:/etc/nginx/ssl/star_iriusrisk_com.crt"
- "./key.pem:/etc/nginx/ssl/star_iriusrisk_com.key"
How should I go around solving this issue? I have looked at "linking" pods with each other but none of the solutions I tried worked. I am bit new to this hence I am still a bit confused about how to expose pods and connect to each other.
The kompose tool now includes the ability to convert to Helm charts from docker-compose.yml files:
kompose convert -c
Check out the kompose Alternative Conversions documentation (also here).
From my current knowledge, there is no such tool is developed or published that converts helm-chart into docker-compose file. But the conversion from docker-compose to kubernetes resource manifests can be done by using tool like kompose (https://kompose.io).
I think it is not necessary to convert from helm chart to docker-compose. You can use Minikube to run whatever is needed locally. Otherwise, the other alternative is to run your containers locally and reverse engineer. i.e. produce the docker compose file. Here is a link to GitHub that does this for you https://github.com/Red5d/docker-autocompose.
Good Luck

How do I ensure same environment for all my workers(containers) in Airflow?

I have a config for deploying 4 pods(hence, 4 workers) for Airflow on Kubernetes using Docker. However, all of a sudden, worker-0 is unable to make a certain curl request whereas other workers are able to make one. This is resulting in the failure of pipelines.
I have tried reading about mismatching configs and stateful sets but in my case, there is one config for all the workers and this is the only single source of truth.
statefulsets-workers.yaml file is as follows:
# Workers are not in deployment, but in StatefulSet, to allow each worker expose a mini-server
# that only serve logs, that will be used by the web server.
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: {{ template "airflow.fullname" . }}-worker
labels:
app: {{ template "airflow.name" . }}-worker
chart: {{ template "airflow.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
serviceName: "{{ template "airflow.fullname" . }}-worker"
updateStrategy:
type: RollingUpdate
# Use experimental burst mode for faster StatefulSet scaling
# https://github.com/kubernetes/kubernetes/commit/****
podManagementPolicy: Parallel
replicas: {{ .Values.celery.num_workers }}
template:
metadata:
{{- if .Values.airflow.pallet.config_path }}
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
{{- end }}
labels:
app: {{ template "airflow.name" . }}-worker
release: {{ .Release.Name }}
spec:
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 1002
fsGroup: 1002
containers:
- name: {{ .Chart.Name }}-worker
imagePullPolicy: {{ .Values.airflow.image_pull_policy }}
image: "{{ .Values.airflow.image }}:{{ .Values.airflow.imageTag }}"
volumeMounts:
{{- if .Values.airflow.storage.enabled }}
- name: google-cloud-key
mountPath: /var/secrets/google
readOnly: true
{{- end }}
- name: worker-logs
mountPath: /usr/local/airflow/logs
- name: data
mountPath: /usr/local/airflow/rootfs
env:
{{- if .Values.airflow.storage.enabled }}
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
{{- end }}
{{- range $setting, $option := .Values.airflow.config }}
- name: {{ $setting }}
value: {{ $option }}
{{- end }}
securityContext:
allowPrivilegeEscalation: false
envFrom:
- configMapRef:
name: pallet-env-file
args: ["worker"]
ports:
- name: wlog
containerPort: 8793
protocol: TCP
{{- if .Values.airflow.image_pull_secret }}
imagePullSecrets:
- name: {{ .Values.airflow.image_pull_secret }}
{{- end }}
{{- if .Values.airflow.storage.enabled }}
volumes:
- name: google-cloud-key
secret:
secretName: {{ .Values.airflow.storage.secretName }}
{{- end }}
volumeClaimTemplates:
- metadata:
name: worker-logs
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Gi
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Gi
I expect all the workers to be able to connect to the service to which I am making the curl request.
It turns out that the environment was indeed the same, however the receiving machine didn't have the new IP of the node whitelisted.
When all the pods crashed, they took the node down with them and restarting the node gave it a new IP. Hence, connection timed out for the worker in that node.

Resources