CrashloopBackOff on Pod in Kubernetes(on GCP with Jenkins) - docker

My pods are under the state of "CrashloopBackOff", the setup is Jenkins with Kubernetes on GCP.
I have found a few answers where it indicates that my Dockerfile is not good and that it needs to be in an infinite state.
But I run the command in the production.yaml ["sh", "-c", "app -port=8080"] to have it in that state.
The exact same Dockerfile was used and it was working when I deployed the project manually to kubernetes.
The project I'm trying to submit looks like this:
The Dockerfile
FROM php:7.2.4-apache
COPY apache_default /etc/apache2/sites-available/000-default.conf
RUN a2enmod rewrite
COPY src /var/www/html/src
COPY public /var/www/html/public
COPY config /var/www/html/config
ADD composer.json /var/www/html
ADD composer.lock /var/www/html
# Install software
RUN apt-get update && apt-get install -y git
# Install unzip
RUN apt-get install -y unzip
# Install curl
RUN apt-get install -y curl
# Install dependencies
RUN php -r "readfile('http://getcomposer.org/installer');" | php -- --install-dir=/usr/bin/ --filename=composer
RUN cd /var/www/html && composer install --no-dev --no-interaction --optimize-autoloader
# install pdo for mysql
RUN docker-php-ext-install pdo pdo_mysql
COPY "memory-limit-php.ini" "/usr/local/etc/php/conf.d/memory-limit-php.ini"
RUN chmod 777 -R /var/www
# Production envivorment
ENV ENVIVORMENT=prod
EXPOSE 80
CMD apachectl -D FOREGROUND
CMD ["app"]
The Jenkinsfile
def project = '****'
def appName = 'wobbl-mobile-backend'
def imageTag = "gcr.io/${project}/${appName}"
def feSvcName = "wobbl-main-backend-service"
pipeline {
agent {
kubernetes {
label 'sample-app'
defaultContainer 'jnlp'
yamlFile 'k8s/pod/pod.yaml'
}
}
stages {
// Deploy Image and push with image container builder
stage('Build and push image with Container Builder') {
steps {
container('gcloud') {
sh "PYTHONUNBUFFERED=1 gcloud container builds submit -t ${imageTag} ."
}
}
}
// Deploy to production
stage('Deploy Production') {
// Production branch
steps{
container('kubectl') {
// Change deployed image in canary to the one we just built
sh("sed -i.bak 's#gcr.io/cloud-solutions-images/wobbl-main:1.0.0#${imageTag}#' ./k8s/production/*.yaml")
sh("kubectl --namespace=production apply -f k8s/services/")
sh("kubectl --namespace=production apply -f k8s/production/")
sh("echo http://`kubectl --namespace=production get service/${feSvcName} -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` > ${feSvcName}")
}
}
}
}
}
Than the yaml kubernetes configurations:
pod.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
component: ci
spec:
# Use service account that can deploy to all namespaces
serviceAccountName: default
containers:
- name: gcloud
image: gcr.io/cloud-builders/gcloud
command:
- cat
tty: true
- name: kubectl
image: gcr.io/cloud-builders/kubectl
command:
- cat
tty: true
The service used backend.yaml
kind: Service
apiVersion: v1
metadata:
name: wobbl-main-backend-service
spec:
ports:
- name: http
port: 8080
targetPort: 8080
protocol: TCP
selector:
role: backend
app: wobbl-main
The deployment production.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: wobbl-main-backend-production
spec:
replicas: 1
template:
metadata:
name: backend
labels:
app: wobbl-main
role: backend
env: production
spec:
containers:
- name: backend
image: gcr.io/cloud-solutions-images/wobbl-main:1.0.0
resources:
limits:
memory: "500Mi"
cpu: "100m"
imagePullPolicy: Always
readinessProbe:
httpGet:
path: /healthz
port: 8080
command: ["sh", "-c", "app -port=8080"]
ports:
- name: backend
containerPort: 8080
When I run kubernetes describe pod **** -n production I get the following response:
Normal Created 3m (x4 over 4m) kubelet,
gke-jenkins-cd-default-pool-83e2f18e-hvwp Created container Normal
Started 3m (x4 over 4m) kubelet,
gke-jenkins-cd-default-pool-83e2f18e-hvwp Started container Warning
BackOff 2m (x8 over 4m) kubelet,
gke-jenkins-cd-default-pool-83e2f18e-hvwp Back-off restarting failed
container
Any hints on how to debug this?

First your Docker file says :
CMD ["app"]
And then within your deployment definition you have :
command: ["sh", "-c", "app -port=8080"]
This is repetition. I suggest you use one of these.
Secondly I assume one of the install commands get you the app binary. Make sure its part of your $PATH
Plus you have a pod and a deployment manifest. I hope you're using either one of them and not deploying both.

Related

Package installed in dockerfile inaccessable in manifest file

I'm quite new to kubernetes and docker.
I am trying to create a kubernetes CronJob which will, every x minutes, clone a repo, build the docker file in that repo, then apply the manifest file to create the job.
When I install git in the CronJob dockerfile, when I run any git command in the kubernetes manifest file, it doesn't recognise it. How should I go about fixing this please?
FROM python:3.8.10
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y git
RUN useradd -rm -d /home/worker -s /bin/bash -g root -G sudo -u 1001 worker
WORKDIR /home/worker
COPY . /home/worker
RUN chown -R 1001:1001 .
USER worker
ENTRYPOINT ["/bin/bash"]
apiVersion: "batch/v1"
kind: CronJob
metadata:
name: cron-job-test
namespace: me
spec:
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox:1.28
imagePullPolicy: Always
command:
- /bin/sh
- -c
args:
- git log;
restartPolicy: OnFailure
You should use the correct image that has git binary installed to run git commands. In the manifest you are using image: busybox:1.28 to run the pod which doesnt have git installed. Hence you are getting the error.
Use correct image name and try

Kubernetes warning for clashes while using env variable in docker? why?

We're using Gitlab for CI/CD. I'll include the script which we're using
gitlab ci-cd file
services:
- docker:19.03.11-dind
before_script:
- apk update && apk add bash
- apk update && apk add gettext
workflow:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "developer" || $CI_COMMIT_BRANCH == "stage"|| ($CI_COMMIT_BRANCH =~ (/^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i))
when: always
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH != "developer" || $CI_COMMIT_BRANCH != "stage"|| ($CI_COMMIT_BRANCH !~ (/^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i))
when: never
stages:
- build
- Publish
- deploy
cache:
paths:
- .m2/repository
- target
build_jar:
image: maven:3.8.3-jdk-11
stage: build
script:
- mvn clean install package -DskipTests=true
artifacts:
paths:
- target/*.jar
docker_build_dev:
stage: Publish
image: docker:19.03.11
services:
- docker:19.03.11-dind
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
only:
- /^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i
- developer
docker_build_stage:
stage: Publish
image: docker:19.03.11
services:
- docker:19.03.11-dind
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
only:
- stage
deploy_dev:
stage: deploy
image: stellacenter/aws-helm-kubectl
variables:
ENV_VAR_NAME: development
before_script:
- aws configure set aws_access_key_id ${DEV_AWS_ACCESS_KEY_ID}
- aws configure set aws_secret_access_key ${DEV_AWS_SECRET_ACCESS_KEY}
- aws configure set region ${DEV_AWS_DEFAULT_REGION}
script:
- sed -i "s/<VERSION>/${CI_COMMIT_SHORT_SHA}/g" patient-service.yml
- mkdir -p $HOME/.kube
- cp $KUBE_CONFIG_DEV $HOME/.kube/config
- chown $(id -u):$(id -g) $HOME/.kube/config
- export KUBECONFIG=$HOME/.kube/config
- cat patient-service.yml | envsubst | kubectl apply -f patient-service.yml -n ${KUBE_NAMESPACE_DEV}
only:
- /^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i
- developer
deploy_stage:
stage: deploy
image: stellacenter/aws-helm-kubectl
variables:
ENV_VAR_NAME: stage
before_script:
- aws configure set aws_access_key_id ${DEV_AWS_ACCESS_KEY_ID}
- aws configure set aws_secret_access_key ${DEV_AWS_SECRET_ACCESS_KEY}
- aws configure set region ${DEV_AWS_DEFAULT_REGION}
script:
- sed -i "s/<VERSION>/${CI_COMMIT_SHORT_SHA}/g" patient-service.yml
- mkdir -p $HOME/.kube
- cp $KUBE_CONFIG_STAGE $HOME/.kube/config
- chown $(id -u):$(id -g) $HOME/.kube/config
- export KUBECONFIG=$HOME/.kube/config
- cat patient-service.yml | envsubst | kubectl apply -f patient-service.yml -n ${KUBE_NAMESPACE_STAGE}
only:
- stage
According to the script, we just merged the script not to face conflicts/clashes for stage and development while deployment. Previously, we having each docker files for each environment(stage and developer). Now I want to merge the dockerfile also, I merged, but the dockerfile is not fetching. Having clashes (warning shows after pipeline succeeds) in Kubernetes. I don't know how to clear the warning in Kubernetes. I'll enclose the docker file which I merged.
FROM maven:3.8.3-jdk-11 AS MAVEN_BUILD
COPY pom.xml /build/
COPY src /build/src/
WORKDIR /build/
RUN mvn clean install package -DskipTests=true
FROM openjdk:11
ARG environment_name
WORKDIR /app
COPY --from=MAVEN_BUILD /build/target/patient-service-*.jar /app/patient-service.jar
ENV PORT 8094
ENV env_var_name=$environment_name
EXPOSE $PORT
ENTRYPOINT ["java","-Dspring.profiles.active= $env_var_name","-jar","/app/patient-service.jar"]
the last line, we used before,
ENTRYPOINT ["java","-Dspring.profiles.active=development","-jar","/app/patient-service.jar"] -for developer dockerfile
ENTRYPOINT ["java","-Dspring.profiles.active=stage","-jar","/app/patient-service.jar"] - for stage dockerfile
At the time, its working fine, I'm not facing any issue on Kubernetes. I'd just add environment variable to fetch along with whether development or stage. You can check ,my script after the docker build. After adding the variable only, we began facing the clashes. Please help me to sort this out. Thanks in advance.
Yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: patient-app
labels:
app: patient-app
spec:
replicas: 1
selector:
matchLabels:
app : patient-app
template:
metadata:
labels:
app: patient-app
spec:
containers:
- name: patient-app
image: registry.gitlab.com/stella-center/backend-services/patient-service:<VERSION>
imagePullPolicy: Always
ports:
- containerPort: 8094
imagePullSecrets:
- name: gitlab-registry-token-auth
---
apiVersion: v1
kind: Service
metadata:
name: patient-service
spec:
type: NodePort
selector:
app: patient-app
ports:
- port: 8094
targetPort: 8094
As I understood you want to run the same image built from this docker file in both environments using the variable in the docker file, I would suggest following below:
1- remove "ENV env_var_name=$environment_name" and add ENV_VAR_NAME directly in the ENTRYPOIT (make sure the variable is upper case) as below .
ENV PORT 8094
EXPOSE $PORT
ENTRYPOINT ["java","-Dspring.profiles.active= $ENV_VAR_NAME","-jar","/app/patient-service.jar"]
2- Add this variable as an environment variable in patient-service.yml:
...
apiVersion: apps/v1
kind: Deployment
metadata:
name: patient-app
labels:
app: patient-app
spec:
replicas: 1
selector:
matchLabels:
app : patient-app
template:
metadata:
labels:
app: patient-app
spec:
containers:
- name: patient-app
image: registry.gitlab.com/stella-center/backend-services/patient-service:<VERSION>
imagePullPolicy: Always
ports:
- containerPort: 8094
env:
- name: ENV_VAR_NAME
value: "${ENV_VAR_NAME}"
imagePullSecrets:
- name: gitlab-registry-token-auth
3- specify the variable in the GitLab ci yml file each stage with its value and use envsubst with the deployment command:
deploy_dev:
stage: deploy
image: stellacenter/aws-helm-kubectl
variables:
ENV_VAR_NAME: development
before_script:
- apk update && apk add gettext
..
script:
..
- cat patient-service.yml | envsubst | kubectl apply -f -n ${KUBE_NAMESPACE_STAGE} -
...

Openshift - Run a basic container with alpine, java and jmeter

In an Openshift environment (Kubernetes v1.18.3+47c0e71)
I am trying to run a very basic container which will contain:
Alpine (latest version)
JDK 1.8
Jmeter 5.3
I just want it to boot and run in a container, expecting connections to run Jmeter CLI from the command line terminal.
I have gotten this to work perfectly in my local Docker distribution. This is the Dokerfile content:
FROM alpine:latest
ARG JMETER_VERSION="5.3"
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV JMETER_DOWNLOAD_URL https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz
USER root
ARG TZ="Europe/Amsterdam"
RUN apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update openjdk8-jre tzdata curl unzip bash \
&& apk add --no-cache nss \
&& rm -rf /var/cache/apk/ \
&& mkdir -p /tmp/dependencies \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} > /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz \
&& mkdir -p /opt \
&& tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt \
&& rm -rf /tmp/dependencies
# Set global PATH such that "jmeter" command is found
ENV PATH $PATH:$JMETER_BIN
WORKDIR ${JMETER_HOME}
For some reason, when I configure a Pod with a container with that exact configuration previously uploaded to a private Docker images registry, it does not work.
This is the Deployment configuration (yaml) file (very basic aswell):
apiVersion: apps/v1
kind: Deployment
metadata:
name: jmeter
namespace: myNamespace
labels:
app: jmeter
group: myGroup
spec:
selector:
matchLabels:
app: jmeter
replicas: 1
template:
metadata:
labels:
app: jmeter
spec:
containers:
- name: jmeter
image: myprivateregistry.azurecr.io/jmeter:dev
resources:
limits:
cpu: 100m
memory: 500Mi
requests:
cpu: 100m
memory: 500Mi
imagePullPolicy: Always
restartPolicy: Always
imagePullSecrets:
- name: myregistrysecret
Unfortunately, I am not getting any logs:
A screenshot of the Pod events:
Unfortunately, not getting either to access the terminal of the container:
Any idea on:
how to get further logs?
what is going on?
On your local machine, you are likely using docker run -it <my_container_image> or similar. Using the -it option will run an interactive shell in your container without you specifying a CMD and will keep that shell running as the primary process started in your container. So by using this command, you are basically already specifying a command.
Kubernetes expects that the container image contains a process that is run on start (CMD) and that will run as long as the container is alive (for example a webserver).
In your case, Kubernetes is starting the container, but you are not specifying what should happen when the container image is started. This leads to the container immediately terminating, which is what you can see in the Events above. Because you are using a Deployment, the failing Pod is then restarted again and again.
A possible workaround to this is to run the sleep command in your container on startup by specifing a command in your Pod like so:
apiVersion: v1
kind: Pod
metadata:
name: command-demo
labels:
purpose: demonstrate-command
spec:
containers:
- name: command-demo-container
image: alpine
command: ["/bin/sleep", "infinite"]
restartPolicy: OnFailure
(Kubernetes documentation)
This will start the Pod and immediately run the /bin/sleep infinite command, leading to the primary process being this sleep process that will never terminate. Your container will now run indefinitely. Now you can use oc rsh <name_of_the_pod to connect to the container and run anything you would like interactively (for example jmeter).

Is there a way to update Jenkins running in Kubernetes?

I'm trying to run Jenkins in Kubernetes but the version of Jenkins is outdated. It says I need atleast version 2.138.4 for the Kubernetes plugin.
Im using this jenkins image from Docker hub ("jenkins/jenkins:lts"). But when I try to run this in Kubernetes it says the version is 2.60.3. I previously used a really old version of Jenkins (2.60.3) but I updated my Dockerfile to use the latest image. After that I build the image again and threw it to Kubernetes. I even delete my Kubernetes Deployment and Service before deploying them again.
I'm currently working in a development environment using Minikube.
Dockerfile:
FROM jenkins/jenkins:lts
ENV JENKINS_USER admin
ENV JENKINS_PASS admin
# Skip initial setup
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
USER root
RUN apt-get update \
&& apt-get install -qqy apt-transport-https ca-certificates curl gnupg2 software-properties-common
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
RUN apt-get update -qq \
&& apt-get install docker-ce -y
RUN usermod -aG docker jenkins
RUN apt-get clean
RUN curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose
USER jenkins
The Kubernetes deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: mikemanders/my-jenkins-image:1.0
env:
- name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
ports:
- name: http-port
containerPort: 8080
- name: jnlp-port
containerPort: 50000
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-home
emptyDir: {}
And the Kubernetes Service:
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
type: NodePort
selector:
app: jenkins
ports:
- port: 8080
targetPort: 8080
I think my Kubernetes configuration are good, so I'm guessing it has something to do with Docker?
What am I missing/doing wrong here?
TL;DR
To update a deployment, you need new Docker image based on the new Jenkins release:
docker build -t mikemanders/my-jenkins-image:1.1 .
docker push mikemanders/my-jenkins-image
kubectl set image deployment/jenkins mikemanders/my-jenkins-image=1.1 --record
Kubernetes deploys images not dockerfiles
As per Images man
You create your Docker image and push it to a registry before referring to it in a Kubernetes pod.
The image property of a container supports the same syntax as the docker command does, including private registries and tags.
So, you need an image to deploy.
Update your image
To update your image in registry, use docker build -t and docker push:
docker build -t mikemanders/my-jenkins-image:1.1
docker push mikemanders/my-jenkins-image
It will rebuild the image with updated jenkins/jeinkis:lts. Then image will be uploaded to the container registry.
The catch is that you are updating the image version (e.g. 1.0->1.1) before updating the cluster.

Concurrent access to docker.sock on k8s

I would like to ask you for a help/advice with the following issue. We are using Bamboo as our CI and we have remote bamboo agents running on k8s.
In our build we have step that creates a Docker image when tests ran correctly. To remote bamboo agents we are exposing Docker via docker.socket. When we had only one remote bamboo agent (to test how it works) everything was working correctly but recently we have increased the number of remote agents. Now it happen quite oft that a build gets stuck in docker image build step and will not move. We have to stop the build and run it again. Usually in logs is no useful info, but once in while this will appear.
24-May-2017 16:04:54 Execution failed for task ':...'.
24-May-2017 16:04:54 > Docker execution failed
24-May-2017 16:04:54 Command line [docker build -t ...] returned:
24-May-2017 16:04:54 time="2017-05-24T16:04:54+02:00" level=info msg="device or resource busy"
This how our k8s deployment looks like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: bamboo-agent
namespace: backend-ci
spec:
replicas: 5
template:
metadata:
labels:
app: bamboo-agent
spec:
containers:
- name: bamboo-agent
stdin: true
resources:
.
env:
.
.
.
ports:
- .
volumeMounts:
- name: dockersocket
mountPath: /var/run/docker.sock
volumes:
- hostPath:
path: /var/run/docker.sock
name: dockersocket
And here is Dockerfile for remote bamboo agent.
FROM java:8
ENV CI true
RUN apt-get update && apt-get install -yq curl && apt-get -yqq install docker.io && apt-get install tzdata -yq
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && mv kubectl /usr/local/bin
RUN echo $TZ | tee /etc/timezone
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN wget -O /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64
RUN chmod +x /usr/local/bin/dumb-init
ADD run.sh /root
ADD .dockercfg /root
ADD config /root/.kube/
ADD config.json /root/.docker/
ADD gradle.properties /root/.gradle/
ADD bamboo-capabilities.properties /root
ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]
CMD /root/run.sh
Is there some way how to solve this issue? And is exposing docker.socket a good solution or is there some better approach?
I have read few articles about Docker in docker but I do not like --privileged mode.
If you need some other information I will try to provide them.
Thank you.
One of the things you can do is run your builds on rkt while running kubernetes on docker?

Resources