Is it possible to to run a container from a gitlab pipeline on a gcp instance which has GPU? I know that I do
> steps:
> - name: 'gcr.io/cloud-builders/docker'
> args: [ 'build', '--memory-swap', '-1', '-t', 'eu.gcr.io/$PROJECT_ID/${_IMAGE}', '.' ]
> - name: 'gcr.io/cloud-builders/docker'
> args: [ 'push', 'eu.gcr.io/$PROJECT_ID/${_IMAGE}']
> - name: "gcr.io/cloud-builders/docker"
> args: ['run', '--rm', \
> '--gpus', 'all', \
> 'eu.gcr.io/$PROJECT_ID/${_IMAGE}', 'pytest']
It fails step 2 with error:
[Gcloud docker --push or pull](docker: Error response from daemon:
could not select device driver "" with capabilities: [[gpu]].)
Basically I need to test the container with a machine with GPU, since the company gitlab runner does not support GPU
Related
We're using Gitlab for CI/CD. I'll include the script which we're using gitlab ci-cd file
services:
- docker:19.03.11-dind
workflow:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "developer" || $CI_COMMIT_BRANCH == "stage"|| ($CI_COMMIT_BRANCH =~ (/^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i))
when: always
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH != "developer" || $CI_COMMIT_BRANCH != "stage"|| ($CI_COMMIT_BRANCH !~ (/^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i))
when: never
stages:
- build
- Publish
- deploy
cache:
paths:
- .m2/repository
- target
build_jar:
image: maven:3.8.3-jdk-11
stage: build
script:
- mvn clean install package -DskipTests=true
artifacts:
paths:
- target/*.jar
docker_build_dev:
stage: Publish
image: docker:19.03.11
services:
- docker:19.03.11-dind
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
only:
- /^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i
- developer
docker_build_stage:
stage: Publish
image: docker:19.03.11
services:
- docker:19.03.11-dind
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
only:
- stage
deploy_dev:
stage: deploy
image: stellacenter/aws-helm-kubectl
variables:
ENV_VAR_NAME: development
before_script:
- apt update
- apt-get install gettext-base
- aws configure set aws_access_key_id ${DEV_AWS_ACCESS_KEY_ID}
- aws configure set aws_secret_access_key ${DEV_AWS_SECRET_ACCESS_KEY}
- aws configure set region ${DEV_AWS_DEFAULT_REGION}
script:
- sed -i "s/<VERSION>/${CI_COMMIT_SHORT_SHA}/g" patient-service.yml
- mkdir -p $HOME/.kube
- cp $KUBE_CONFIG_DEV $HOME/.kube/config
- chown $(id -u):$(id -g) $HOME/.kube/config
- export KUBECONFIG=$HOME/.kube/config
- cat patient-service.yml | envsubst | kubectl apply -f patient-service.yml -n ${KUBE_NAMESPACE_DEV}
only:
- /^([A-Z]([0-9][-_])?)?SPRINT(([-_][A-Z][0-9])?)+/i
- developer
deploy_stage:
stage: deploy
image: stellacenter/aws-helm-kubectl
variables:
ENV_VAR_NAME: stage
before_script:
- apt update
- apt-get install gettext-base
- aws configure set aws_access_key_id ${DEV_AWS_ACCESS_KEY_ID}
- aws configure set aws_secret_access_key ${DEV_AWS_SECRET_ACCESS_KEY}
- aws configure set region ${DEV_AWS_DEFAULT_REGION}
script:
- sed -i "s/<VERSION>/${CI_COMMIT_SHORT_SHA}/g" patient-service.yml
- mkdir -p $HOME/.kube
- cp $KUBE_CONFIG_STAGE $HOME/.kube/config
- chown $(id -u):$(id -g) $HOME/.kube/config
- export KUBECONFIG=$HOME/.kube/config
- cat patient-service.yml | envsubst | kubectl apply -f patient-service.yml -n ${KUBE_NAMESPACE_STAGE}
only:
- stage
According to the script, we just merged the script not to face conflicts/clashes for stage and development enviornment while deployment. Previously, we having each docker files for each environment(stage and developer). Now I want to merge the dockerfile & k8's yml file also, I merged, but the dockerfile is not fetching. Having clashes (its showing the warning message "back-off restarting failed container"after pipeline succeeds) in Kubernetes. I don't know how to clear the warning in Kubernetes. I'll enclose the docker file and yml file for your reference which I merged.
k8's yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: patient-app
labels:
app: patient-app
spec:
replicas: 1
selector:
matchLabels:
app : patient-app
template:
metadata:
labels:
app: patient-app
spec:
containers:
- name: patient-app
image: registry.gitlab.com/stella-center/backend-services/patient-service:<VERSION>
imagePullPolicy: Always
ports:
- containerPort: 8094
env:
- name: ENV_VAR_NAME
value: "${ENV_VAR_NAME}"
imagePullSecrets:
- name: gitlab-registry-token-auth
---
apiVersion: v1
kind: Service
metadata:
name: patient-service
spec:
type: NodePort
selector:
app: patient-app
ports:
- port: 8094
targetPort: 8094
Docker file
FROM maven:3.8.3-jdk-11 AS MAVEN_BUILD
COPY pom.xml /build/
COPY src /build/src/
WORKDIR /build/
RUN mvn clean install package -DskipTests=true
FROM openjdk:11
WORKDIR /app
COPY --from=MAVEN_BUILD /build/target/patient-service-*.jar /app/patient-service.jar
ENV PORT 8094
EXPOSE $PORT
ENTRYPOINT ["java","-Dspring.profiles.active=$ENV_VAR_NAME","-jar","/app/patient-service.jar"]
In dockerfile , before we used the last line, we used before,
ENTRYPOINT ["java","-Dspring.profiles.active=development","-jar","/app/patient-service.jar"] -for developer dockerfile
ENTRYPOINT ["java","-Dspring.profiles.active=stage","-jar","/app/patient-service.jar"] - for stage dockerfile
At the time, its working fine, I'm not facing any issue on Kubernetes. I just added environment variable to fetch along with whether development or stage .I don't know why the warning is happening. Please help me to sort this out . Thanks in advance.
kubectl describe pods
> Name: patient-app-6cd8c88d6-s7ldt Namespace:
> stellacenter-dev Priority: 0 Node:
> ip-192-168-49-35.us-east-2.compute.internal/192.168.49.35 Start Time:
> Wed, 25 May 2022 20:09:23 +0530 Labels: app=patient-app
> pod-template-hash=6cd8c88d6 Annotations: kubernetes.io/psp: eks.privileged Status: Running IP:
> 192.168.50.146 IPs: IP: 192.168.50.146 Controlled By: ReplicaSet/patient-app-6cd8c88d6 Containers: patient-app:
> Container ID: docker://2d3431a015a40f551e51285fa23e1d39ad5b257bfd6ba75c3972f422b94b12be
> Image: registry.gitlab.com/stella-center/backend-services/patient-service:96e21d80
> Image ID: docker-pullable://registry.gitlab.com/stella-center/backend-services/patient-service#sha256:3f9774efe205c081de4df5b6ee22cba9940f974311b094
> 2a8473ee02b9310b43
> Port: 8094/TCP
> Host Port: 0/TCP
> State: Running
> Started: Wed, 25 May 2022 20:09:24 +0530
> Ready: True
> Restart Count: 0
> Environment: <none>
> Mounts:
> /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sxbzc (ro) Conditions: Type Status
> Initialized True Ready True ContainersReady
> True PodScheduled True Volumes: kube-api-access-sxbzc:
> Type: Projected (a volume that contains injected data from multiple sources)
> TokenExpirationSeconds: 3607
> ConfigMapName: kube-root-ca.crt
> ConfigMapOptional: <nil>
> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations:
> node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
> node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>
Your Dockerfile uses exec form ENTRYPOINT syntax. This form doesn't expand environment variables; Spring is literally getting the string $ENV_VAR_NAME as the profile name, and failing on this.
Spring knows how to set properties from environment variables, though. Rather than building that setting into the Dockerfile, you can use an environment variable to set the profile name at deploy time.
# Dockerfile: do not set `-Dspring.profiles.active`
ENTRYPOINT ["java", "-jar", "/app/patient-service.jar"]
# Deployment YAML: do set `$SPRING_PROFILES_ACTIVE`
env:
- name: SPRING_PROFILES_ACTIVE
value: "${ENV_VAR_NAME}" # Helm: {{ quote .Values.environment }}
However, with this approach, you still need to set deployment-specific settings in your src/main/resources/application-*.yml file, then rebuild the jar file, then rebuild the Docker image, then redeploy. This doesn't make sense for most settings, particularly since you can set them as environment variables. If one of these values needs to change you can just change the Kubernetes configuration and redeploy, without recompiling anything.
# Deployment YAML: don't use Spring profiles; directly set variables instead
env:
- name: SPRING_DATASOURCE_URL
value: "jdbc:postgresql://postgres-dev/database"
Run the following command to get the output of why your pod crashes:
kubectl describe pod -n <your-namespace> <your-pod>.
Additionally the output of kubectl get pod -o yaml -n <your-namespace> <your-pod> has a status section that holds the reason for restarts. You might have to lookup the exit code. E.g. 137 stands for OOM.
Currently the cloudbuild.yaml looks like this:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args: [ 'artifacts', 'docker', 'images', 'delete', 'location-docker.pkg.dev/$PROJECT_ID/repository/image' ]
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'location-docker.pkg.dev/$PROJECT_ID/repository/image:latest', './' ]
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'location-docker.pkg.dev/$PROJECT_ID/reporitory/image:latest']
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args: ['run', 'deploy', 'image', '--image', 'us-east1-docker.pkg.dev/$PROJECT_ID/cb-area52-ar/area52-image:latest', '--region', 'region']
images:
- 'location-docker.pkg.dev/$PROJECT_ID/registry/image:latest'
That does basically the following:
delete the existsing image in the artifact registry
build the new image
pushes it back to the artifact registry
deploys it to google cloud run
My problem is now that the first step fails whenever there is no image in the registry.
How can i prevent it from cancelling the whole build process when this occurs?
You can create an inline script to check whether the image exists or not. This assumes you always want to delete the image with the latest tag.
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-eEuo'
- 'pipefail'
- '-c'
- |-
if [[ -z `gcloud artifacts docker images describe location-docker.pkg.dev/$PROJECT_ID/repository/image:latest --verbosity=none --format=text` ]]
then
echo "Image does not exist. Continue with the build"
else
echo "Deleting Image"
gcloud artifacts docker images delete location-docker.pkg.dev/$PROJECT_ID/repository/image
fi
I'm trying to create a Windows Docker container using Kaniko/Gitlab.
Here is the Error I see:
Resolving secrets
00:00
Preparing the "docker-windows" executor
Using Docker executor with image gcr.io/kaniko-project/executor:v1.6.0-debug ...
Pulling docker image gcr.io/kaniko-project/executor:v1.6.0-debug ...
WARNING: Failed to pull image with policy "always": no matching manifest for windows/amd64 10.0.17763 in the manifest list entries (docker.go:147:0s)
ERROR: Preparation failed: failed to pull image "gcr.io/kaniko-project/executor:v1.6.0-debug" with specified policies [always]: no matching manifest for windows/amd64 10.0.17763 in the manifest list entries (docker.go:147:0s)
For .gitlab-ci.yml file:
image:
name: microsoft/iis:latest
entrypoint: [""]
.build_variables: &build_variables
TAG: "docker-base-windows-2019-std-core"
AWS_ACCOUNT: "XXXXXXXXXX"
AWS_REGION: "XXXXXXX"
REGISTRY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
.build_script: &build_script
script:
- echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $REGISTRY:$TAG
stages:
- build-docker-image
build_image_dev:
variables:
<<: *build_variables
stage: build-docker-image
image:
name: gcr.io/kaniko-project/executor:v1.6.0-debug
entrypoint: [""]
tags: ['XXXXX']
<<: *build_script
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH == "main"'
- if: $CI_COMMIT_TAG
This is normal text Code for Docker file:
FROM Microsoft/iis:latest
CMD [ "cmd" ]
You have the error:
no matching manifest for windows/amd64
which means that particular image could not be found. It happens if you develop on windows and your server is a linux for instance.
This error implies your host machine's OS is not compatible with the OS docker image you are trying to pull.
I'm having a problem with Google Cloud Build where the docker build command doesn't seem to be accepting the build-arg option, even though this same command works as expected on local:
Dockerfile:
ARG ASSETS_ENV=development
RUN echo "ASSETS_ENV is ${ASSETS_ENV}"
Build Command:
docker build --build-arg="ASSETS_ENV=production" .
Result on local:
ASSETS_ENV is production
Result on Cloud Build:
ASSETS_ENV is development
Ok the fix was in the cloud build yaml config:
Before:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '--build-arg="ASSETS_ENV=production"', '.']
After:
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: ['-c', 'docker build --build-arg="ASSETS_ENV=production" .']
For anyone who defines their steps like this:
- name: 'gcr.io/cloud-builders/docker'
args:
- build
- foo
- bar
The comment from #nader-ghanbari worked for me:
- name: 'gcr.io/cloud-builders/docker'
args:
- build
- --build-arg
- TF2_BASE_IMAGE=${_TF2_BASE_IMAGE}
I have a scenario, I run my Jenkins in K8s cluster in Minikube. I run a groovy script within my Jenkins Pipeline which builds the docker image
using Kaniko (which builds a docker image without docker daemon) and pushes to Azure container registry. I have created secret to authenticate to Azure.
But when I push an image - I get an error
" [36mINFO[0m[0004] Taking snapshot of files...
[36mINFO[0m[0004] ENTRYPOINT ["jenkins-slave"]
error pushing image: failed to push to destination Testimage.azurecr.io/test:latest: unexpected end of JSON input
[Pipeline] }"
My Script
My groovy script --
def label = "kaniko-${UUID.randomUUID().toString()}"
podTemplate(name: 'kaniko', label: label, yaml: """
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
volumeMounts:
- name: jenkins-pv
mountPath: /root
volumes:
- name: jenkins-pv
projected:
sources:
- secret:
name: pass
items:
- key: .dockerconfigjson
path: .docker/config.json
"""
) {
node(label) {
stage('Build with Kaniko') {
git 'https://github.com/jenkinsci/docker-jnlp-slave.git'
container(name: 'kaniko', shell: '/busybox/sh') {
sh '''#!/busybox/sh
/kaniko/executor -f `pwd`/Dockerfile -c `pwd` --skip-tls-verify --destination=testimage.azurecr.io/test:latest
'''
}
}
}
}
Could you please help how to overcome the error? And also :
How do I know the name of the image build by kaiko?
I m just pushing like - registry.acr.io/test: latest, probably it's an incorrect image name that's the reason I get JSON output error?