Unable to build Docker images through Jenkins installed on Kubernetes - docker

Question

I used the following helm chart to install Jenkins
https://artifacthub.io/packages/helm/jenkinsci/jenkins
The problem is it does't build docker images, saying there's no docker. Docker was installed on host with sudo apt install docker-ce docker-ce-cli containerd.io
var/jenkins_home/workspace/test#tmp/durable-23187d38/script.sh: 2: /var/jenkins_home/workspace/test#tmp/durable-23187d38/script.sh: docker: not found
There was also another error
Caused: java.io.IOException: Cannot run program "docker": error=2, No such file or directory
I tried experimenting with multiple values.yaml variations with no luck.
Here's the latest one(I removed the ingress part). There might be redundant parts like that podTemplates: with docker config(copied it from some github repo) and agent: volumes: where I mount the same thing /var/run/docker.sock I don't understand where it should go.
I'm using k3s if it matters. And this is just a plain VPS.
# Default values for jenkins.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value
## Overrides for generated resource names
# See templates/_helpers.tpl
# nameOverride:
# fullnameOverride:
# namespaceOverride:
# For FQDN resolving of the controller service. Change this value to match your existing configuration.
# ref: https://github.com/kubernetes/dns/blob/master/docs/specification.md
clusterZone: "cluster.local"
renderHelmLabels: true
controller:
# Used for label app.kubernetes.io/component
componentName: "jenkins-controller"
image: "jenkins/jenkins"
tag: "2.277.2-jdk11"
imagePullPolicy: "Always"
imagePullSecretName:
# Optionally configure lifetime for controller-container
lifecycle:
# postStart:
# exec:
# command:
# - "uname"
# - "-a"
disableRememberMe: false
numExecutors: 1
# configures the executor mode of the Jenkins node. Possible values are: NORMAL or EXCLUSIVE
executorMode: "NORMAL"
# This is ignored if enableRawHtmlMarkupFormatter is true
markupFormatter: plainText
customJenkinsLabels: []
# The default configuration uses this secret to configure an admin user
# If you don't need that user or use a different security realm then you can disable it
adminSecret: true
hostNetworking: false
# When enabling LDAP or another non-Jenkins identity source, the built-in admin account will no longer exist.
# If you disable the non-Jenkins identity store and instead use the Jenkins internal one,
# you should revert controller.adminUser to your preferred admin user:
adminUser: "admin"
# adminPassword: <defaults to random>
admin:
existingSecret: ""
userKey: jenkins-admin-user
passwordKey: jenkins-admin-password
# This values should not be changed unless you use your custom image of jenkins or any devired from. If you want to use
# Cloudbees Jenkins Distribution docker, you should set jenkinsHome: "/var/cloudbees-jenkins-distribution"
jenkinsHome: "/var/jenkins_home"
# This values should not be changed unless you use your custom image of jenkins or any devired from. If you want to use
# Cloudbees Jenkins Distribution docker, you should set jenkinsRef: "/usr/share/cloudbees-jenkins-distribution/ref"
jenkinsRef: "/usr/share/jenkins/ref"
# Path to the jenkins war file which is used by jenkins-plugin-cli.
jenkinsWar: "/usr/share/jenkins/jenkins.war"
resources:
# requests:
# cpu: "50m"
# memory: "256Mi"
limits:
cpu: "2000m"
memory: "2048Mi"
# Environment variables that get added to the init container (useful for e.g. http_proxy)
# initContainerEnv:
# - name: http_proxy
# value: "http://192.168.64.1:3128"
# containerEnv:
# - name: http_proxy
# value: "http://192.168.64.1:3128"
# Set min/max heap here if needed with:
# javaOpts: "-Xms512m -Xmx512m"
# jenkinsOpts: ""
# If you are using the ingress definitions provided by this chart via the `controller.ingress` block the configured hostname will be the ingress hostname starting with `https://` or `http://` depending on the `tls` configuration.
# The Protocol can be overwritten by specifying `controller.jenkinsUrlProtocol`.
# jenkinsUrlProtocol: "https"
# If you are not using the provided ingress you can specify `controller.jenkinsUrl` to change the url definition.
# jenkinsUrl: ""
# If you set this prefix and use ingress controller then you might want to set the ingress path below
# jenkinsUriPrefix: "/jenkins"
# Enable pod security context (must be `true` if podSecurityContextOverride, runAsUser or fsGroup are set)
usePodSecurityContext: true
# Note that `runAsUser`, `fsGroup`, and `securityContextCapabilities` are
# being deprecated and replaced by `podSecurityContextOverride`.
# Set runAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
# When setting runAsUser to a different value than 0 also set fsGroup to the same value:
runAsUser: 1000
fsGroup: 1000
# If you have PodSecurityPolicies that require dropping of capabilities as suggested by CIS K8s benchmark, put them here
securityContextCapabilities: {}
# drop:
# - NET_RAW
# Completely overwrites the contents of the `securityContext`, ignoring the
# values provided for the deprecated fields: `runAsUser`, `fsGroup`, and
# `securityContextCapabilities`. In the case of mounting an ext4 filesystem,
# it might be desirable to use `supplementalGroups` instead of `fsGroup` in
# the `securityContext` block: https://github.com/kubernetes/kubernetes/issues/67014#issuecomment-589915496
# podSecurityContextOverride:
# runAsUser: 1000
# runAsNonRoot: true
# supplementalGroups: [1000]
# # capabilities: {}
servicePort: 8080
targetPort: 8080
# For minikube, set this to NodePort, elsewhere use LoadBalancer
# Use ClusterIP if your setup includes ingress controller
serviceType: ClusterIP
# Jenkins controller service annotations
serviceAnnotations: {}
# Jenkins controller custom labels
statefulSetLabels: {}
# foo: bar
# bar: foo
# Jenkins controller service labels
serviceLabels: {}
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
# Put labels on Jenkins controller pod
podLabels: {}
# Used to create Ingress record (should used with ServiceType: ClusterIP)
# nodePort: <to set explicitly, choose port between 30000-32767
# Enable Kubernetes Liveness and Readiness Probes
# if Startup Probe is supported, enable it too
# ~ 2 minutes to allow Jenkins to restart when upgrading plugins. Set ReadinessTimeout to be shorter than LivenessTimeout.
healthProbes: true
probes:
startupProbe:
httpGet:
path: '{{ default "" .Values.controller.jenkinsUriPrefix }}/login'
port: http
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 12
livenessProbe:
failureThreshold: 5
httpGet:
path: '{{ default "" .Values.controller.jenkinsUriPrefix }}/login'
port: http
periodSeconds: 10
timeoutSeconds: 5
# If Startup Probe is not supported on your Kubernetes cluster, you might want to use "initialDelaySeconds" instead.
# It delays the initial liveness probe while Jenkins is starting
# initialDelaySeconds: 60
readinessProbe:
failureThreshold: 3
httpGet:
path: '{{ default "" .Values.controller.jenkinsUriPrefix }}/login'
port: http
periodSeconds: 10
timeoutSeconds: 5
# If Startup Probe is not supported on your Kubernetes cluster, you might want to use "initialDelaySeconds" instead.
# It delays the initial readyness probe while Jenkins is starting
# initialDelaySeconds: 60
agentListenerPort: 50000
agentListenerHostPort:
agentListenerNodePort:
disabledAgentProtocols:
- JNLP-connect
- JNLP2-connect
csrf:
defaultCrumbIssuer:
enabled: true
proxyCompatability: true
# Kubernetes service type for the JNLP agent service
# agentListenerServiceType is the Kubernetes Service type for the JNLP agent service,
# either 'LoadBalancer', 'NodePort', or 'ClusterIP'
# Note if you set this to 'LoadBalancer', you *must* define annotations to secure it. By default
# this will be an external load balancer and allowing inbound 0.0.0.0/0, a HUGE
# security risk: https://github.com/kubernetes/charts/issues/1341
agentListenerServiceType: "ClusterIP"
# Optionally assign an IP to the LoadBalancer agentListenerService LoadBalancer
# GKE users: only regional static IPs will work for Service Load balancer.
agentListenerLoadBalancerIP:
agentListenerServiceAnnotations: {}
# Example of 'LoadBalancer' type of agent listener with annotations securing it
# agentListenerServiceType: LoadBalancer
# agentListenerServiceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-internal: "True"
# service.beta.kubernetes.io/load-balancer-source-ranges: "172.0.0.0/8, 10.0.0.0/8"
# LoadBalancerSourcesRange is a list of allowed CIDR values, which are combined with ServicePort to
# set allowed inbound rules on the security group assigned to the controller load balancer
loadBalancerSourceRanges:
- 0.0.0.0/0
# Optionally assign a known public LB IP
# loadBalancerIP: 1.2.3.4
# Optionally configure a JMX port
# requires additional javaOpts, ie
# javaOpts: >
# -Dcom.sun.management.jmxremote.port=4000
# -Dcom.sun.management.jmxremote.authenticate=false
# -Dcom.sun.management.jmxremote.ssl=false
# jmxPort: 4000
# Optionally configure other ports to expose in the controller container
extraPorts: []
# - name: BuildInfoProxy
# port: 9000
# List of plugins to be install during Jenkins controller start
# installPlugins:
# - kubernetes:latest
# - kubernetes-credentials-provider:latest
# - workflow-aggregator:2.6
# - git:latest
# - configuration-as-code:1.47
# Set to false to download the minimum required version of all dependencies.
installLatestPlugins: false
# List of plugins to install in addition to those listed in controller.installPlugins
additionalPlugins: []
# Enable to initialize the Jenkins controller only once on initial installation.
# Without this, whenever the controller gets restarted (Evicted, etc.) it will fetch plugin updates which has the potential to cause breakage.
# Note that for this to work, `persistence.enabled` needs to be set to `true`
initializeOnce: false
# Enable to always override the installed plugins with the values of 'controller.installPlugins' on upgrade or redeployment.
# overwritePlugins: true
# Configures if plugins bundled with `controller.image` should be overwritten with the values of 'controller.installPlugins' on upgrade or redeployment.
overwritePluginsFromImage: true
# Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
# The plugin is not installed by default, please update controller.installPlugins.
enableRawHtmlMarkupFormatter: false
# Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
scriptApproval: []
# - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
# - "new groovy.json.JsonSlurperClassic"
# List of groovy init scripts to be executed during Jenkins controller start
initScripts: []
# - |
# print 'adding global pipeline libraries, register properties, bootstrap jobs...'
additionalSecrets: []
# - name: nameOfSecret
# value: secretText
# Below is the implementation of Jenkins Configuration as Code. Add a key under configScripts for each configuration area,
# where each corresponds to a plugin or section of the UI. Each key (prior to | character) is just a label, and can be any value.
# Keys are only used to give the section a meaningful name. The only restriction is they may only contain RFC 1123 \ DNS label
# characters: lowercase letters, numbers, and hyphens. The keys become the name of a configuration yaml file on the controller in
# /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin. The lines after each |
# become the content of the configuration yaml file. The first line after this is a JCasC root element, eg jenkins, credentials,
# etc. Best reference is https://<jenkins_url>/configuration-as-code/reference. The example below creates a welcome message:
JCasC:
defaultConfig: true
configScripts: {}
# welcome-message: |
# jenkins:
# systemMessage: Welcome to our CI\CD server. This Jenkins is configured and managed 'as code'.
# Ignored if securityRealm is defined in controller.JCasC.configScripts and
# ignored if controller.enableXmlConfig=true as controller.securityRealm takes precedence
securityRealm: |-
local:
allowsSignup: false
enableCaptcha: false
users:
- id: "${chart-admin-username}"
name: "Jenkins Admin"
password: "${chart-admin-password}"
# Ignored if authorizationStrategy is defined in controller.JCasC.configScripts
authorizationStrategy: |-
loggedInUsersCanDoAnything:
allowAnonymousRead: false
# Optionally specify additional init-containers
customInitContainers: []
# - name: custom-init
# image: "alpine:3.7"
# imagePullPolicy: Always
# command: [ "uname", "-a" ]
sidecars:
configAutoReload:
# If enabled: true, Jenkins Configuration as Code will be reloaded on-the-fly without a reboot. If false or not-specified,
# jcasc changes will cause a reboot and will only be applied at the subsequent start-up. Auto-reload uses the
# http://<jenkins_url>/reload-configuration-as-code endpoint to reapply config when changes to the configScripts are detected.
enabled: true
image: kiwigrid/k8s-sidecar:0.1.275
imagePullPolicy: IfNotPresent
resources: {}
# limits:
# cpu: 100m
# memory: 100Mi
# requests:
# cpu: 50m
# memory: 50Mi
# How many connection-related errors to retry on
reqRetryConnect: 10
# env:
# - name: REQ_TIMEOUT
# value: "30"
# SSH port value can be set to any unused TCP port. The default, 1044, is a non-standard SSH port that has been chosen at random.
# Is only used to reload jcasc config from the sidecar container running in the Jenkins controller pod.
# This TCP port will not be open in the pod (unless you specifically configure this), so Jenkins will not be
# accessible via SSH from outside of the pod. Note if you use non-root pod privileges (runAsUser & fsGroup),
# this must be > 1024:
sshTcpPort: 1044
# folder in the pod that should hold the collected dashboards:
folder: "/var/jenkins_home/casc_configs"
# If specified, the sidecar will search for JCasC config-maps inside this namespace.
# Otherwise the namespace in which the sidecar is running will be used.
# It's also possible to specify ALL to search in all namespaces:
# searchNamespace:
# Allows you to inject additional/other sidecars
other: []
## The example below runs the client for https://smee.io as sidecar container next to Jenkins,
## that allows to trigger build behind a secure firewall.
## https://jenkins.io/blog/2019/01/07/webhook-firewalls/#triggering-builds-with-webhooks-behind-a-secure-firewall
##
## Note: To use it you should go to https://smee.io/new and update the url to the generete one.
# - name: smee
# image: docker.io/twalter/smee-client:1.0.2
# args: ["--port", "{{ .Values.controller.servicePort }}", "--path", "/github-webhook/", "--url", "https://smee.io/new"]
# resources:
# limits:
# cpu: 50m
# memory: 128Mi
# requests:
# cpu: 10m
# memory: 32Mi
# Name of the Kubernetes scheduler to use
schedulerName: ""
# Node labels and tolerations for pod assignment
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
nodeSelector: {}
terminationGracePeriodSeconds:
tolerations: []
affinity: {}
# Leverage a priorityClass to ensure your pods survive resource shortages
# ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
priorityClassName:
podAnnotations: {}
# Add StatefulSet annotations
statefulSetAnnotations: {}
# StatefulSet updateStrategy
# ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategy: {}
ingress:
# configures the hostname e.g. jenkins.example.com
tls:
- hosts:
- "jen.kirqe.be"
secretName: jen-kirqe-be-tls
# Can be used to disable rendering controller test resources when using helm template
testEnabled: true
agent:
enabled: true
defaultsProviderTemplate: ""
# URL for connecting to the Jenkins contoller
jenkinsUrl:
# connect to the specified host and port, instead of connecting directly to the Jenkins controller
jenkinsTunnel:
kubernetesConnectTimeout: 5
kubernetesReadTimeout: 15
maxRequestsPerHostStr: "32"
namespace: jenkins
image: "jenkins/inbound-agent"
tag: "4.6-1"
workingDir: "/home/jenkins"
customJenkinsLabels: []
# name of the secret to be used for image pulling
imagePullSecretName:
componentName: "jenkins-agent"
websocket: false
privileged: false
runAsUser:
runAsGroup:
resources:
requests:
cpu: "512m"
memory: "512Mi"
limits:
cpu: "512m"
memory: "512Mi"
# You may want to change this to true while testing a new image
alwaysPullImage: false
# Controls how agent pods are retained after the Jenkins build completes
# Possible values: Always, Never, OnFailure
podRetention: "Never"
# You can define the volumes that you want to mount for this container
# Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, PVC, Secret
# Configure the attributes as they appear in the corresponding Java class for that type
# https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
volumes:
# - type: ConfigMap
# configMapName: myconfigmap
# mountPath: /var/myapp/myconfigmap
# - type: EmptyDir
# mountPath: /var/myapp/myemptydir
# memory: false
- type: HostPath
hostPath: /var/run/docker.sock
mountPath: /var/run/docker.sock
# - type: Nfs
# mountPath: /var/myapp/mynfs
# readOnly: false
# serverAddress: "192.0.2.0"
# serverPath: /var/lib/containers
# - type: PVC
# claimName: mypvc
# mountPath: /var/myapp/mypvc
# readOnly: false
# - type: Secret
# defaultMode: "600"
# mountPath: /var/myapp/mysecret
# secretName: mysecret
# Pod-wide environment, these vars are visible to any container in the agent pod
# You can define the workspaceVolume that you want to mount for this container
# Allowed types are: DynamicPVC, EmptyDir, HostPath, Nfs, PVC
# Configure the attributes as they appear in the corresponding Java class for that type
# https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes/workspace
workspaceVolume: {}
# - type: DynamicPVC
# configMapName: myconfigmap
# - type: EmptyDir
# memory: false
# - type: HostPath
# hostPath: /var/lib/containers
# - type: Nfs
# readOnly: false
# serverAddress: "192.0.2.0"
# serverPath: /var/lib/containers
# - type: PVC
# claimName: mypvc
# readOnly: false
# Pod-wide environment, these vars are visible to any container in the agent pod
envVars: []
# - name: PATH
# value: /usr/local/bin
nodeSelector: {}
# Key Value selectors. Ex:
# jenkins-agent: v1
# Executed command when side container gets started
command:
args: "${computer.jnlpmac} ${computer.name}"
# Side container name
sideContainerName: "jnlp"
# Doesn't allocate pseudo TTY by default
TTYEnabled: false
# Max number of spawned agent
containerCap: 10
# Pod name
podName: "default"
# Allows the Pod to remain active for reuse until the configured number of
# minutes has passed since the last step was executed on it.
idleMinutes: 0
# Raw yaml template for the Pod. For example this allows usage of toleration for agent pods.
# https://github.com/jenkinsci/kubernetes-plugin#using-yaml-to-define-pod-templates
# https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
yamlTemplate: ""
# yamlTemplate: |-
# apiVersion: v1
# kind: Pod
# spec:
# tolerations:
# - key: "key"
# operator: "Equal"
# value: "value"
# Defines how the raw yaml field gets merged with yaml definitions from inherited pod templates: merge or override
yamlMergeStrategy: "override"
# Timeout in seconds for an agent to be online
connectTimeout: 100
# Annotations to apply to the pod.
annotations: {}
# Below is the implementation of custom pod templates for the default configured kubernetes cloud.
# Add a key under podTemplates for each pod template. Each key (prior to | character) is just a label, and can be any value.
# Keys are only used to give the pod template a meaningful name. The only restriction is they may only contain RFC 1123 \ DNS label
# characters: lowercase letters, numbers, and hyphens. Each pod template can contain multiple containers.
# For this pod templates configuration to be loaded the following values must be set:
# controller.JCasC.defaultConfig: true
# Best reference is https://<jenkins_url>/configuration-as-code/reference#Cloud-kubernetes. The example below creates a python pod template.
podTemplates:
docker: |
- name: docker
label: docker
containers:
- name: docker
image: docker:latest
command: "/bin/sh -c"
args: "cat"
ttyEnabled: true
privileged: true
resourceRequestCpu: "400m"
resourceRequestMemory: "512Mi"
resourceLimitCpu: "1"
resourceLimitMemory: "1024Mi"
volumes:
- hostPathVolume:
hostPath: /var/run/docker.sock
mountPath: /var/run/docker.sock
- hostPathVolume:
hostPath: /data
mountPath: /data
# Here you can add additional agents
# They inherit all values from `agent` so you only need to specify values which differ
additionalAgents: {}
# maven:
# podName: maven
# customJenkinsLabels: maven
# # An example of overriding the jnlp container
# # sideContainerName: jnlp
# image: jenkins/jnlp-agent-maven
# tag: latest
# python:
# podName: python
# customJenkinsLabels: python
# sideContainerName: python
# image: python
# tag: "3"
# command: "/bin/sh -c"
# args: "cat"
# TTYEnabled: true
persistence:
enabled: true
## A manually managed Persistent Volume and Claim
## Requires persistence.enabled: true
## If defined, PVC must be created manually before volume will be bound
existingClaim:
## jenkins data Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
storageClass:
annotations: {}
accessMode: "ReadWriteOnce"
size: "5Gi"
volumes:
# - name: jenkins
# persistentVolumeClaim:
# claimName: jenkins-claim
mounts:
# - name: jenkins
# mountPath: /var/jenkins_home
# volumes:
# # - name: nothing
# # emptyDir: {}
# mounts:
# # - mountPath: /var/nothing
# # name: nothing
# # readOnly: true
networkPolicy:
# Enable creation of NetworkPolicy resources.
enabled: false
# For Kubernetes v1.4, v1.5 and v1.6, use 'extensions/v1beta1'
# For Kubernetes v1.7, use 'networking.k8s.io/v1'
apiVersion: networking.k8s.io/v1
# You can allow agents to connect from both within the cluster (from within specific/all namespaces) AND/OR from a given external IP range
internalAgents:
allowed: true
podLabels: {}
namespaceLabels: {}
# project: myproject
externalAgents: {}
# ipCIDR: 172.17.0.0/16
# except:
# - 172.17.1.0/24
## Install Default RBAC roles and bindings
rbac:
create: true
readSecrets: false
serviceAccount:
create: true
# The name of the service account is autogenerated by default
name:
annotations: {}
imagePullSecretName:
serviceAccountAgent:
# Specifies whether a ServiceAccount should be created
create: false
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name:
annotations: {}
imagePullSecretName:
checkDeprecation: true

Answer 1

You are running Jenkins itself as a container. Therefore the docker command line application must be present in the container, not the host.
Easiest solution: Use a Jenkins docker image that contains the docker cli already, for example https://hub.docker.com/r/trion/jenkins-docker-client

Answer 2

I would suggest using Kaniko in your pipeline as it's more secure and following best practices.
an example pipeline:
pipeline {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
metadata:
labels:
jenkins: worker
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
command: ["/busybox/cat"]
tty: true
volumeMounts:
- name: dockercred
mountPath: /root/.docker/
volumes:
- name: dockercred
secret:
secretName: dockercred
"""
}
}
stages {
stage('Stage 1: Build with Kaniko') {
steps {
container('kaniko') {
sh '/kaniko/executor --context=git://github.com/repository/project.git \
--destination=repository/image:tag \
--insecure \
--skip-tls-verify \
-v=debug'
}
}
}
}
}
you will need to create a docker credential file and mount it as a configmap/secret in order to push the image.