How to create tekton task for building a docker slim? - docker

Has anyone tried to create tekton task on building a docker slim?

Turns out we can re-use the "dind-sidecar" sample from the tekton pipelines repository:
https://github.com/tektoncd/pipeline/blob/main/examples/v1alpha1/taskruns/dind-sidecar.yaml
I got it working using the following:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: docker-build-dockerslim
spec:
params:
- default: docker.io/dslim/docker-slim:latest
description: The location of the kaniko builder image.
name: builderimage
type: string
- default: docker.io/docker:stable
description: The location of the Docker builder image.
name: pushimage
type: string
- default: "registry.default.svc.cluster.local:5000"
description: When using an insecure (for push/pull to a non-TLS registry), we should set its name here. Don't set an empty string, remove option from task or set it to a dummy value if not required.
name: insecure
type: string
- default: docker.io/docker:dind
description: The location of the Docker in Docker image.
name: dindimage
type: string
- default: Dockerfile
description: The name of the Dockerfile
name: dockerfile
type: string
- default: .
description: Parent directory for your Dockerfile.
name: dockerroot
type: string
resources:
inputs:
- name: source
type: git
outputs:
- name: image
type: image
steps:
- args:
- --state-path
- /dslim-state
- --in-container
- build
- --http-probe=false
- --dockerfile
- $(inputs.params.dockerfile)
- --dockerfile-context
- $(inputs.params.dockerroot)
- $(outputs.resources.image.url)
env:
- name: DOCKER_HOST
value: tcp://127.0.0.1:2376
- name: DOCKER_TLS_VERIFY
value: '1'
- name: DOCKER_CERT_PATH
value: /certs/client
image: $(inputs.params.builderimage)
name: build
resources: {}
securityContext:
privileged: true
volumeMounts:
- mountPath: /dslim-state
name: state
- mountPath: /certs/client
name: dind-certs
workingDir: /workspace/source
- command:
- /bin/sh
- -c
- |
SLIM_IMAGE=$(docker images | awk '/docker-slim.*[0-9]*\.slim/{print $1;exit 0;}')
docker tag "$SLIM_IMAGE" $(outputs.resources.image.url)
docker push $(outputs.resources.image.url)
name: push
image: $(params.pushimage)
env:
- name: DOCKER_HOST
value: tcp://127.0.0.1:2376
- name: DOCKER_TLS_VERIFY
value: '1'
- name: DOCKER_CERT_PATH
value: /certs/client
volumeMounts:
- mountPath: /certs/client
name: dind-certs
sidecars:
- args:
- --storage-driver=vfs
- --userland-proxy=false
- --debug
- --insecure-registry=$(inputs.params.insecure)
env:
- name: DOCKER_TLS_CERTDIR
value: /certs
image: $(inputs.params.dindimage)
name: dind
readinessProbe:
periodSeconds: 1
exec:
command:
- ls
- /certs/client/ca.pem
resources: {}
securityContext:
privileged: true
volumeMounts:
- mountPath: /certs/client
name: dind-certs
volumes:
- name: dind-certs
emptyDir: {}
- emptyDir: {}
name: state
For some reason, the resulting images doesn't appear with the name I expected it to. Also tried setting a "--target" argument, though neither this nor the default "last-arg-is-image-name" behavior they document in their readme seems to work ( https://github.com/docker-slim/docker-slim ).
However I did find, listing images, the following:
docker-slim-tmp-fat-image.12.20211205135050.slim latest 0037ff15e1f5 2 seconds ago 13.8MB
docker-slim-empty-image latest 9dfd57fb50a8 35 seconds ago 0B
docker-slim-tmp-fat-image.12.20211205135050 latest 9ad36dd5e3f3 39 seconds ago 211MB
<Dockerfiles-FROM-image> master f11e63190556 3 months ago 211MB
Thus, I do some "docker images | awk ..." then "docker tag xxx.slim the-target-name-I-wanted", before my "docker push".
Resulting image is indeed smaller. I'ld to test this with other images, and make sure it doesn't introduce any regression, ... still, that's interesting.

Related

What is kubeflow gpu resource node allocation criteria?

I’m curious about the Kubeflow GPU Resource. I’m running the job below.
The only part where I specified the GPU Resource is on first container with only 1 GPU. However, the event message tells me 0/4 nodes are available: 4 Insufficient nvidia.com/gpu.
Why is this job searching for 4 nodes though I specified only 1 GPU resource? Does my interpretation have a problem? Thanks much in advance.
FYI) I have 3 worker nodes with each 1 gpu.
apiVersion: batch/v1
kind: Job
metadata:
name: saint-train-3
annotations:
sidecar.istio.io/inject: "false"
spec:
template:
spec:
initContainers:
- name: dataloader
image: <AWS CLI Image>
command: ["/bin/sh", "-c", "aws s3 cp s3://<Kubeflow Bucket>/kubeflowdata.tar.gz /s3-data; cd /s3-data; tar -xvzf kubeflowdata.tar.gz; cd kubeflow_data; ls"]
volumeMounts:
- mountPath: /s3-data
name: s3-data
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef: {key: AWS_ACCESS_KEY_ID, name: aws-secret}
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef: {key: AWS_SECRET_ACCESS_KEY, name: aws-secret}
containers:
- name: trainer
image: <Our Model Image>
command: ["/bin/sh", "-c", "wandb login <ID>; python /opt/ml/src/main.py --base_path='/s3-data/kubeflow_data' --debug_mode='0' --project='kubeflow-test' --name='test2' --gpu=0 --num_epochs=1 --num_workers=4"]
volumeMounts:
- mountPath: /s3-data
name: s3-data
resources:
limits:
nvidia.com/gpu: "1"
- name: gpu-watcher
image: pytorch/pytorch:latest
command: ["/bin/sh", "-c", "--"]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- mountPath: /s3-data
name: s3-data
volumes:
- name: s3-data
persistentVolumeClaim:
claimName: test-claim
restartPolicy: OnFailure
backoffLimit: 6
0/4 nodes are available: 4 Insufficient nvidia.com/gpu
This is mean you haven't nodes with label nvidia.com/gpu

Error in windows docker desktop kubernetes

In kubernetes on windows docker desktop when I try to mount an empty directory I get the following error:
error: error when retrieving current configuration of:
Resource: "/v1, Resource=pods", GroupVersionKind: "/v1, Kind=Pod"
Name: "", Namespace: "default"
Object: &{map["apiVersion":"v1" "kind":"Pod" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "namespace":"default"] "spec":map["containers":[map["image":"nginx:alpine" "name":"nginx" "volumeMounts":[map["mountPath":"/usr/share/nginx/html" "name":"html" "readOnly":%!q(bool=true)]]] map["args":["while true; do date >> /html/index.html; sleep 10; done"] "command":["/bin/sh" "-c"] "image":"alpine" "name":"html-updater" "volumeMounts":[map["mountPath":"/html" "name":"html"]]]] "volumes":[map["emptyDir":map[] "name":"html"]]]]}
from server for: "nginx-alpine-emptyDir.pod.yml": resource name may not be empty
The error message seems a bit unclear and I cannot find what's going on.
My yaml configuration is the following:
apiVersion: v1
kind: Pod
spec:
volumes:
- name: html
emptyDir: {}
containers:
- name: nginx
image: nginx:alpine
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
readOnly: true
- name: html-updater
image: alpine
command: ["/bin/sh", "-c"]
args:
- while true; do date >> /html/index.html; sleep 10; done
volumeMounts:
- name: html
mountPath: /html
Forgot to add metadata name
metadata:
name: empty-dir-test
Code after change is:
apiVersion: v1
kind: Pod
metadata:
name: empty-dir-test
spec:
volumes:
- name: html
emptyDir: {}
containers:
- name: nginx
image: nginx:alpine
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
readOnly: true
- name: html-updater
image: alpine
command: ["/bin/sh", "-c"]
args:
- while true; do date >> /html/index.html; sleep 10; done
volumeMounts:
- name: html
mountPath: /html

kubernetes jenkins plugin does not create 2 containers

I have 2 Jenkins instances, one use version 1.8 and second version 1.18.
Oldest version is able to create both containers.
Agent specification [Kubernetes Pod Template] (mo-aio-build-supplier):
* [jnlp] mynexus.services.com/mo-base/jenkins-slave-mo-aio:1.8.2-ca(resourceRequestCpu: 0.25, resourceRequestMemory: 256Mi, resourceLimitCpu: 1, resourceLimitMemory: 1.5Gi)
* [postgres] mynexus.services.com:443/mo-base/mo-base-postgresql-95-openshift
Newest version are not able to create postgres container
Container postgres exited with error 1. Logs: mkdir: cannot create directory '/home/jenkins': Permission denied
Both use same podTemplate
podTemplate(
name: label,
label: label,
cloud: 'openshift',
serviceAccount: 'jenkins',
containers: [
containerTemplate(
name: 'jnlp',
image: 'mynexus.services.theosmo.com/jenkins-slave-mo-aio:v3.11.104-14_jdk8',
resourceRequestCpu: env.CPU_REQUEST,
resourceLimitCpu: env.CPU_LIMIT,
resourceRequestMemory: env.RAM_REQUEST,
resourceLimitMemory: env.RAM_LIMIT,
workingDir: '/tmp',
args: '${computer.jnlpmac} ${computer.name}',
command: ''
),
containerTemplate(
name: 'postgres',
image: 'mynexus.services.theosmo.com:443/mo-base/mo-base-postgresql-95-openshift',
envVars: [
envVar(key: "POSTGRESQL_USER", value: "admin"),
envVar(key: "POSTGRESQL_PASSWORD", value: "admin"),
envVar(key: "POSTGRESQL_DATABASE", value: "supplier_data"),
]
)
],
volumes: [emptyDirVolume(mountPath: '/dev/shm', memory: true)]
)
Also, I've noticed YAML created by newest version is a bit weird
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins.svc:80/job/build-supplier/473/"
labels:
jenkins: "slave"
jenkins/mo-aio-build-supplier: "true"
name: "mo-aio-build-supplier-xfgmn-qmrdl"
spec:
containers:
- args:
- "********"
- "mo-aio-build-supplier-xfgmn-qmrdl"
env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "jenkins-jnlp.svc:50000"
- name: "JENKINS_AGENT_NAME"
value: "mo-aio-build-supplier-xfgmn-qmrdl"
- name: "JENKINS_NAME"
value: "mo-aio-build-supplier-xfgmn-qmrdl"
- name: "JENKINS_AGENT_WORKDIR"
value: "/tmp"
- name: "JENKINS_URL"
value: "http://jenkins.svc:80/"
- name: "HOME"
value: "/home/jenkins"
image: "mynexus.services.com/mo-base/jenkins-slave-mo-aio:1.8.2-ca"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources:
limits:
memory: "1.5Gi"
cpu: "1"
requests:
memory: "256Mi"
cpu: "0.25"
securityContext:
privileged: false
tty: false
volumeMounts:
- mountPath: "/dev/shm"
name: "volume-0"
readOnly: false
- mountPath: "/tmp"
name: "workspace-volume"
readOnly: false
workingDir: "/tmp"
- env:
- name: "POSTGRESQL_DATABASE"
value: "supplier_data"
- name: "POSTGRESQL_USER"
value: "admin"
- name: "HOME"
value: "/home/jenkins"
- name: "POSTGRESQL_PASSWORD"
value: "admin"
image: "mynexus.services.com:443/mo-base/mo-base-postgresql-95-openshift"
imagePullPolicy: "IfNotPresent"
name: "postgres"
resources:
limits: {}
requests: {}
securityContext:
privileged: false
tty: false
volumeMounts:
- mountPath: "/dev/shm"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
nodeSelector: {}
restartPolicy: "Never"
serviceAccount: "jenkins"
volumes:
- emptyDir:
medium: "Memory"
name: "volume-0"
- emptyDir: {}
name: "workspace-volume"
As you are able to see above:
postgres container is under an env tree
Any suggestion? Thanks in advance
As far as I checked there
The problem
Since Kubernetes Plugin version 1.18.0, the default working directory of the pod containers was changed from /home/jenkins to /home/jenkins/agent. But the default HOME environment variable enforcement is still pointing to /home/jenkins. The impact of this change is that if pod container images do not have a /home/jenkins directory with sufficient permissions for the running user, builds will fail to do anything directly under their HOME directory, /home/jenkins.
Resolution
There are different workaround to that problem:
Change the default HOME variable
The simplest and preferred workaround is to add the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent on Jenkins startup. This requires a restart.
This workaround will reflect the behavior of kubernetes plugin pre-1.18.0 but on the new working directory /home/jenkins/agent
Use /home/jenkins as the working directory
A workaround is to change the working directory of pod containers back to /home/jenkins. This workaround is only possible when using YAML to define agent pod templates (see JENKINS-60977).
Prepare images for Jenkins
A workaround could be to ensure that the images used in agent pods have a /home/jenkins directory that is owned by the root group and writable by the root group as mentioned in OpenShift Container Platform-specific guidelines.
Additionaly there is the issue on jenkins.
Hope this helps.

error resolving dockerfile path: please provide a valid path to a Dockerfile within the build context with --dockerfile

apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args:
- "--context=dir:///workspace"
- "--dockerfile=/workspace/Dockerfile"
- "--destination=gcr.io/kubernetsjenkins/jenkinsondoc:latest"
volumeMounts:
- name: kaniko-secret
mountPath: /secret
- name: context
mountPath: /workspace
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /secret/kaniko-secret.json
restartPolicy: Never
volumes:
- name: kaniko-secret
secret:
secretName: kaniko-secret
- name: context
hostPath:
path: /home/sabadsulla/kanikodir
I am running kaniko on a kubernetes pod to build a docker image and pushing to the GCR.
When i use google cloud storage for the CONTEXT_PATH it works fine ,
But i need to use the Local_directory(meaning using the shared volumes of the pods) AS the CONTEXT_PATH
it throws an error
"Error: error resolving dockerfile path: please provide a valid path to a Dockerfile within the build context with --dockerfile
Usage:
I tried with args "--context=/workspace" , "--context=dir://workspace" , it gives the same error
the folder looks like
In host:
/home/sabadsulla/kanikodir/Dockerfile
When it turns to PV/PVC, in pod container
/workspace/Dockerfile
Then for kanino executor, if we map the context to workspace, the dockerfile will be related to context is Dockerfile, so
--context=/workspace
--dockerfile=Dockerfile
using kaniko container and volume mounted as persistent volume claim.
Please try and use"--dockerfile=./Dockerfile"
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args: ["--dockerfile=./Dockerfile",
"--context=/workspace/",
"--destination=gcr.io/kubernetsjenkins/jenkinsondoc:latest"]
volumeMounts:
- name: kaniko-secret
mountPath: /secret
- name: context
mountPath: /workspace/
Using the default values:
--dockerfile string -Path to the dockerfile to be built. (default "Dockerfile")
--context string -Path to the dockerfile build context. (default "/workspace/")
Even this one statement works:
args: ["--destination=gcr.io/kubernetsjenkins/jenkinsondoc:latest"]
Hope this help. Could you please test it and share with the results?.
hi i just solved this problem.
my node name: m1.env.lab.io
my Dockerfile path: /root/kaniko/demo1/Dockerfile
FROM ubuntu
ENTRYPOINT ["/bin/bash", "-c", "echo hello"]
pod.yaml in /root/kaniko/demo1/pod.yaml:
apiVersion: v1
kind: Pod
metadata:
name: kaniko
namespace: kaniko
spec:
nodeName: m1.env.lab.io
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args: ["--verbosity=trace",
"--log-format=color",
"--dockerfile=Dockerfile",
"--context=dir:///workspace/",
"--destination=registry.local/cloud2go/kaniko-ubuntu:v0.1"] # no account and password for my registry.
volumeMounts:
- name: dockerfile-storage
mountPath: /workspace/
restartPolicy: Never
volumes:
- name: dockerfile-storage
hostPath:
path: /root/kaniko/demo1

How to clone a private git repository into a kubernetes pod using ssh keys in secrets?

I am trying to clone a private git repository(gitLab) into a kubernetes pod, using SSH keys for authentication. I have stored my keys in a secret. Here is the yaml file for the job that does the desired task.
Heres the same question, but doesnt give the exact solution :
Clone a secure git repo in Kubernetes pod
Logs of the init container after execution:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
v3.7.1-66-gfc22ab4fd3 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.1-55-g7d5f104fa7 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
OK: 9064 distinct packages available
OK: 23 MiB in 23 packages
Cloning into '/tmp'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
The yaml file which works perfectly for public repo:
apiVersion: batch/v1
kind: Job
metadata:
name: nest-build-kaniko
labels:
app: nest-kaniko-example
spec:
template:
spec:
containers:
-
image: 'gcr.io/kaniko-project/executor:latest'
name: kaniko
args: ["--dockerfile=/workspace/Dockerfile",
"--context=/workspace/",
"--destination=aws.dest.cred"]
volumeMounts:
-
mountPath: /workspace
name: source
-
name: aws-secret
mountPath: /root/.aws/
-
name: docker-config
mountPath: /kaniko/.docker/
initContainers:
-
name: download
image: alpine:3.7
command: ["/bin/sh","-c"]
args: ['apk add --no-cache git && git clone https://github.com/username/repo.git /tmp/']
volumeMounts:
-
mountPath: /tmp
name: source
restartPolicy: Never
volumes:
-
emptyDir: {}
name: source
-
name: aws-secret
secret:
secretName: aws-secret
-
name: docker-config
configMap:
name: docker-config
The yaml file after using git-sync for cloning private repository:
apiVersion: batch/v1
kind: Job
metadata:
name: nest-build-kaniko
labels:
app: nest-kaniko-example
spec:
template:
spec:
containers:
-
image: 'gcr.io/kaniko-project/executor:latest'
name: kaniko
args: ["--dockerfile=/workspace/Dockerfile",
"--context=/workspace/",
"--destination=aws.dest.cred"]
volumeMounts:
-
mountPath: /workspace
name: source
-
name: aws-secret
mountPath: /root/.aws/
-
name: docker-config
mountPath: /kaniko/.docker/
initContainers:
-
name: git-sync
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
volumeMounts:
-
mountPath: /git/tmp
name: source
-
name: git-secret
mountPath: "/etc/git-secret"
env:
- name: GIT_SYNC_REPO
value: "git#gitlab.com:username/repo.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_DEST
value: "/tmp"
- name: GIT_SYNC_ONE_TIME
value: "true"
securityContext:
runAsUser: 0
restartPolicy: Never
volumes:
-
emptyDir: {}
name: source
-
name: aws-secret
secret:
secretName: aws-secret
-
name: git-secret
secret:
secretName: git-creds
defaultMode: 256
-
name: docker-config
configMap:
name: docker-config
You can use git-sync
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: git-sync-test
spec:
selector:
matchLabels:
app: git-sync-test
serviceName: "git-sync-test"
replicas: 1
template:
metadata:
labels:
app: git-sync-test
spec:
containers:
- name: git-sync-test
image: <your-main-image>
volumeMounts:
- name: service
mountPath: /var/magic
initContainers:
- name: git-sync
image: k8s.gcr.io/git-sync-amd64:v2.0.6
imagePullPolicy: Always
volumeMounts:
- name: service
mountPath: /magic
- name: git-secret
mountPath: /etc/git-secret
env:
- name: GIT_SYNC_REPO
value: <repo-path-you-want-to-clone>
- name: GIT_SYNC_BRANCH
value: <repo-branch>
- name: GIT_SYNC_ROOT
value: /magic
- name: GIT_SYNC_DEST
value: <path-where-you-want-to-clone>
- name: GIT_SYNC_PERMISSIONS
value: "0777"
- name: GIT_SYNC_ONE_TIME
value: "true"
- name: GIT_SYNC_SSH
value: "true"
securityContext:
runAsUser: 0
volumes:
- name: service
emptyDir: {}
- name: git-secret
secret:
defaultMode: 256
secretName: git-creds # your-ssh-key
For more details check this link.
initContainers:
-
name: git-sync
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
volumeMounts:
-
mountPath: /workspace
name: source
-
name: git-secret
mountPath: "/etc/git-secret"
env:
- name: GIT_SYNC_REPO
value: "git#gitlab.com:username/repo.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_ROOT
value: /workspace
- name: GIT_SYNC_DEST
value: "tmp"
- name: GIT_SYNC_ONE_TIME
value: "true"
NOTE: set GIT_SYNC_ROOT env to /workspace
It'll clone in /workspace/tmp directory in your emptyDir source volume.

Resources