I have 2 programs that need to be run periodically.
simple_job_1.py:
from datetime import datetime
import time
print("Starting job 1 ... ", datetime.now())
print("Doing stuff in job 1 for 20 seconds .........")
time.sleep(20)
print("Stopping job 1 ... ", datetime.now())
simple_job_2.py:
from datetime import datetime
import time
print("Starting job 2 ... ", datetime.now())
print("Doing stuff in job 2 for 5 seconds .........")
time.sleep(5)
print("Stopping job 2 ... ", datetime.now())
And I have created 2 docker images by building following Dockerfile's:
For job1:
FROM python:3
# Create a folder inside the container
RUN mkdir /home/TestProj
# Copy everything from the current folder in host to the dest folder in container
COPY . /home/TestProj
WORKDIR /home/TestProj
COPY . .
CMD ["python", "simple_job_1.py"]
For job2:
FROM python:3
# Create a folder inside the container
RUN mkdir /home/TestProj
# Copy everything from the current folder in host to the dest folder in container
COPY . /home/TestProj
WORKDIR /home/TestProj
COPY . .
CMD ["python", "simple_job_2.py"]
Here is how I build these container images:
docker build -t simple_job_1:1.0 .
docker build -t simple_job_2:1.0 .
And here is my docker-compose yaml file:
simple_compose.yaml:
version: "3.9"
services:
job1:
image: simple_job_1:1.0
job2:
image: simple_job_2:1.0
Q1) I need to run this compose - say - every 10th minute as a cronjob. How can I achieve it? I know that it is possible to run containers as cronjobs but is it possible to do that with docker compose?
Q2) Is it possible to run the docker-compose as a cronjob in Google Cloud GKE?
Q3) How can I make sure that job2 only starts after job1 completes?
ADDENDUM:
Here is an example of cronjob spec for running containers as cronjobs in GKE:
# cronjob.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: simple-job-cj-1
spec:
schedule: "0 8 * * *"
concurrencyPolicy: Allow
startingDeadlineSeconds: 100
suspend: false
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
containers:
- name: simple-job-cj-1
image: simple_job
restartPolicy: OnFailure
But please note that this is running a given container. By not being an expert in this field, I guess I can define multiple containers under the containers: section in the spec: above, which probably(?) means I do not need to use docker-compose then?? But if that is the case, how can I make sure that job2 only starts after job1 completes running?
Related
I have a use case that my "./main" binary should run inside the pod and stop after some time (90 seconds) before launching a new pod by the cronJob object.
But I am not confused about how to add both sleep and run my binary in the background together. Please suggest a good approach to this and excuse me for any wrong syntax.
Dockerfile
FROM golang:alpine
WORKDIR /app
COPY main /app
RUN apk update && apk add bash
CMD ["./main &"]
---
cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: cron
namespace: test-cron
spec:
schedule: "*/2 * * * *"
concurrencyPolicy: Replace
successfulJobsHistoryLimit: 0
failedJobsHistoryLimit: 0
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
volumes:
- name: log
hostPath:
path: /data/log/test-cron/
containers:
- name: test-cron
image: test-kafka-0.5
command: ["sleep", "90"] // By adding this, the sleep command is working but my binary is not running inside my container.
Kubernetes has built-in support for killing a Pod after a deadline; you do not need to try to implement this manually.
In your Dockerfile, set up your image to run the program normally. Do not try to include any sort of "sleep" or "kill" option, and do not try to run the program in the background.
CMD ["./main"]
In the Job spec, you need to set an activeDeadlineSeconds: field.
apiVersion: batch/v1
kind: CronJob
spec:
jobTemplate:
spec:
activeDeadlineSeconds: 90 # <-- add
template:
spec:
containers:
- name: test-cron
image: test-kafka-0.5
# do not override `command:`
An identical option exists at the Pod level. For your use case this doesn't especially matter, but if the Job launches multiple Pods then there's a difference of whether each individual Pod has the deadline or whether the Job as a whole does.
It looks from the question like you're trying to run this job every two minutes and not have two concurrent copies of it. Assuming the Pod is created and starts promptly, this should accomplish that. If you had a reason to believe the Job would run faster the second time and just want to restart it, the CronJob might not be the setup you're after.
I have 5 tasks in my project that need to be run periodically. Some of these tasks are run on a daily basis, some on a weekly basis.
I try to containerize each task in a Docker image. Here is one illustrative example:
FROM tensorflow/tensorflow:2.7.0
RUN mkdir /home/MyProject
COPY . /home/MyProject
WORKDIR /home/MyProject/M1/src/
RUN pip install pandas numpy
CMD ./task1.sh
There are a list of Python scripts that need to be run in the task1.sh file defined above. This is not a server application or anything similar, it will run the task1.sh, which will run all the python scripts defined in it one by one, and the entire process will be finished within minutes. And the same process is supposed to be repeated 24 hours later.
How can I schedule such Docker containers in GCP? Are there different ways of doing it? Which one is comparably simpler if there are multiple solutions?
I am not a dev-ops expert by any means. All examples in documentation I find are explained for server applications which are running all the time, not like my example where the image needs to be run just once periodically. This topic is quite daunting for a beginner in this domain like myself.
ADDENDUM:
Looking at Google's documentation for cronjobs in GKE on the following page:
https://cloud.google.com/kubernetes-engine/docs/how-to/cronjobs
I find the following cronjob.yaml file:
# cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: hello
spec:
schedule: "*/1 * * * *"
concurrencyPolicy: Allow
startingDeadlineSeconds: 100
suspend: false
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo "Hello, World!"
restartPolicy: OnFailure
It is stated that this cronjob prints the current time and a string once every minute.
But it is documented in a way with the assumption that you deeply understand what is going on on the page, in which case you would not need to read the documentation!
Let's say that I have my image that I would like it to be run once every day, and the name of my image - say - is my_image.
I assume that I am supposed to change the following part for my own image.
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo "Hello, World!"
It is a total mystery what these names and arguments mean.
name: hello
I suppose it is just a user selected name and does not have any practical importance.
image: busybox
Is this busybox the base image? If not, what is that? It says NOTHING about what this busybox thing is and where it comes from!
args:
- /bin/sh
- -c
- date; echo "Hello, World!"
And based on the explanation on the page, this is the part that prints the date and the "Hello, World!" string to the screen.
Ok... So, how do I modify this template to create a cronjob out of my own image my_image? This documentation does not help at all!
I will answer your comment here, because the second part of your question is too long to answer.
Don't be afraid, it's kubernetes API definition. You declare what you want to the control plane. It is in charge to make your whishes happen!
# cronjob.yaml
apiVersion: batch/v1 # The API that you call
kind: CronJob # The type of object/endpoint in that API
metadata:
name: hello # The name of your job definition
spec:
schedule: "*/1 * * * *" # Your scheduling, change it to "0 10 * * *" to run your job every dat at 10.00am
concurrencyPolicy: Allow # config stuff, deep dive later
startingDeadlineSeconds: 100 # config stuff, deep dive later
suspend: false # config stuff, deep dive later
successfulJobsHistoryLimit: 3 # config stuff, deep dive later
failedJobsHistoryLimit: 1 # config stuff, deep dive later
jobTemplate: # Your execution definition
spec:
template:
spec:
containers:
- name: hello # Custom name of your container. Only to help you in case of debug, logs, ...
image: busybox # Image of your container, can be gcr.io/projectID/myContainer for example
args: # Args to pass to your container. You also have the "entrypoint" definition to change if you want. The entrypoint is the binary to run and that will receive the args
- /bin/sh
- -c
- date; echo "Hello, World!"
# You can also use "command" to run the command with the args directly. In fact it's WHAT you start in your container to perform the job.
restartPolicy: OnFailure # Config in case of failure.
You have more details on the API definition here
Here the API definition of a container with all the possible values to customize it.
I want to create a Jelastic environment with a load balancer and a cp node. I want to add the cp node with the addNodes api method, because it needs specific data to start. My manifest looks like this:
jpsVersion: 1.3
jpsType: install
application:
id: test-app
name: Test App
version: 0.0
env:
topology:
nodes:
- nodeGroup: bl
nodeType: nginx-dockerized
tag: 1.16.1
displayName: Node balancing
count: 1
fixedCloudlets: 1
cloudlets: 4
onInstall:
- addFile
- setup
actions:
addFile:
- cmd [bl]:
- mkdir /data
- echo "Hello world" > /data/test.txt
user: root
setup:
- addNodes:
- nodeGroup: cp
nodeType: docker
displayName: Test Mount
count: 1
fixedCloudlets: 1
cloudlets: 4
dockerName: alpine
volumeMounts:
/kickstart:
readOnly: true
sourcePath: /data
sourceNodeGroup: bl
dockerVolumes:
- /kickstart
For some reason, I want my alpine image to be provided with the data I am storing in the folder /kickstart. Of course, in that case, it's completely irrelevant. The example above is just kept simple enough to be reproducible. In my real use-case, the docker image I want to mount will not be able to run without some application-specific data that are filled up with the manifest's settings completed upon user input. That's why it is necessary that the data be available upon docker node addition.
My problem is that the above simple manifest does not work. Indeed, on my docker node, I have no access to /kickstart/test.txt folder. What am I doing wrong?
volumeMounts option is not available for action addNodes
Here is an example how to implement it:
type: install
name: Test App
nodes:
- nodeGroup: bl
nodeType: nginx
tag: 1.16.1
displayName: Node balancing
cloudlets: 4
- nodeGroup: cp
displayName: Test Mount
cloudlets: 4
image: alpine
startServiceOnCreation: false
volumes:
- /kickstart
volumeMounts:
/kickstart:
sourcePath: /data
sourceNodeGroup: bl
readOnly: true
onInstall:
- cmd [bl]: |-
echo "Hello world" > /data/test.txt
user: root
- env.control.ExecDockerRunCmd [${nodes.cp.join(id,)}]
also you can use non-existent directories in volumeMounts
it will create everything by itself
I am trying to push my container up to GCP Kubernetes in my cluster. My pod runs locally but it doesn't want to run on GCP. It comes back with this error Error response from daemon: No command specified: CreateContainerError
It worked if I run it locally in docker but once I push it up to the container registry on gcp and apply the deployment yaml using kubectl apply -f in my namespace it never brings it up and just keeps saying
gce-exporting-fsab83222-85sc4 0/1 CreateContainerError 0 5m6s
I can't get any logs out of it either:
Error from server (BadRequest): container "gce" in pod "gce-exporting-fsab83222-85sc4" is waiting to start: CreateContainerError
Heres my files below:
Dockerfile:
FROM alpine:3.8
WORKDIR /build
COPY test.py /build
RUN chmod 755 /build/test.py
CMD ["python --version"]
CMD ["python", "test.py"]
Python Script:
#!/usr/bin/python3
import time
def your_function():
print("Hello, World")
while True:
your_function()
time.sleep(10) #make function to sleep for 10 seconds
yaml file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: gce-exporting
namespace: "monitoring"
spec:
selector:
matchLabels:
app: gce
template:
metadata:
labels:
app: gce
spec:
containers:
- name: gce
image: us.gcr.io/lab-manager/lab/gce-exporting:latest
I have tried using CMD and Entrypoint at the end to make sure the pod is running but no luck.
This is the output of the describe pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 60s default-scheduler Successfully assigned monitoring/gce-exporting-fsab83222-85sc4 to gke-dev-lab-standard-8efad9b6-8m66
Normal Pulled 5s (x7 over 59s) kubelet, gke-dev-lab-standard-8efad9b6-8m66 Container image "us.gcr.io/lab-manager/lab/gce-exporting:latest" already present on machine
Warning Failed 5s (x7 over 59s) kubelet, gke-dev-lab-standard-8efad9b6-8m66 Error: Error response from daemon: No command specified
It was a malformed character in my Dockerfile and caused it to crash.
You might need to update your Dockerfile with following:
FROM python
WORKDIR /build
ENV PYTHONUNBUFFERED=1
ENV PYTHONIOENCODING=UTF-8
COPY test.py /build
RUN chmod 755 /build/test.py
CMD ["python", "test.py"]
Then build and push the docker image and recreate pod. Hope it helps!
So Jenkins is installed inside the cluster with this official helm chart. And this is my installed plugins as per helm release values:
installPlugins:
- kubernetes:1.18.1
- workflow-job:2.33
- workflow-aggregator:2.6
- credentials-binding:1.19
- git:3.11.0
- blueocean:1.19.0
my Jenkinsfile relies on the following pod template to spin up slaves:
kind: Pod
spec:
# dnsConfig:
# options:
# - name: ndots
# value: "1"
containers:
- name: dind
image: docker:19-dind
command:
- cat
tty: true
volumeMounts:
- name: dockersock
readOnly: true
mountPath: /var/run/docker.sock
resources:
limits:
cpu: 500m
memory: 512Mi
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
Slaves (pod /dind container) starts nicely as expected whenever there is new Build.
However, it broke at the step of "docker build" in ( Jenkinsfile pipeline
docker build -t ... ) and it breaks there :
Step 16/24 : RUN ../gradlew clean bootJar
---> Running in f14b6418b3dd
Downloading https://services.gradle.org/distributions/gradle-5.5-all.zip
Exception in thread "main" java.net.UnknownHostException: services.gradle.org
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)
at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at org.gradle.wrapper.Download.downloadInternal(Download.java:67)
at org.gradle.wrapper.Download.download(Download.java:52)
at org.gradle.wrapper.Install$1.call(Install.java:62)
at org.gradle.wrapper.Install$1.call(Install.java:48)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)
The command '/bin/sh -c ../gradlew clean bootJar' returned a non-zero code:
At the first galance, I thought it's DNS resolution issue with the Slave container (docker:19-dind) since it is alpine.
That's why I debug its /etc/resolv.conf by adding sh "cat /etc/resolv.conf" in the Jenkinsfile.
I got :
nameserver 172.20.0.10
search cicd.svc.cluster.local svc.cluster.local cluster.local ap-southeast-1.compute.internal
options ndots:5
I removed the last line options ndots:5 as per recommendation of many thread on the internet.
But it does not fix the issue. 😔
I thought again and again and I realized that the container responsible for this error is not the Slave (docker:19-dind), instead, it is the intermediate containers that are opened to satisfy docker build.
As consequence, I added RUN cat /etc/resolv.conf as another layer in the Dockerfile (which starts by FROM gradle:5.5-jdk11).
Now, the resolv.conf is different :
Step 15/24 : RUN cat /etc/resolv.conf
---> Running in 91377c9dd519
; generated by /usr/sbin/dhclient-script
search ap-southeast-1.compute.internal
options timeout:2 attempts:5
nameserver 10.0.0.2
Removing intermediate container 91377c9dd519
---> abf33839df9a
Step 16/24 : RUN ../gradlew clean bootJar
---> Running in f14b6418b3dd
Downloading https://services.gradle.org/distributions/gradle-5.5-all.zip
Exception in thread "main" java.net.UnknownHostException: services.gradle.org
Basically, it is a different nameserver 10.0.0.2 than the nameserver of the slave container 172.20.0.10. There is NO ndots:5 in resolv.conf this intermediate container.
I was confused after all these debugging steps and a lot of attempts.
Architecture
Jenkins Server (Container )
||
(spin up slaves)
||__ SlaveA (Container, image: docker:19-dind)
||
( run "docker build" )
||
||_ intermediate (container, image: gradle:5.5-jdk11 )
Just add --network=host to docker build or docker run.
docker build --network=host foo/bar:latest .
Found the answer here