I want to deploy an application to Airflow that accepts a config file as a parameter, pulls the git repository specified by said config, then transforms it into a Docker image, then uploads that image to GCP's Artifact Registry. What is the best practice for building a docker image inside an Airflow DAG?
I have tried orchestrating a manually-triggered cloud build run via Airflow - I have not been able to pass the necessary substitutions into the cloudbuild.yaml file using the CloudBuildCreateBuildOperator, nor have I been able to specify the workspace.
I have also created a docker image that itself can create new docker images (when the docker.sock file is mounted as a volume). However, using a KubernetesPodOperator to call this seems to go against the design philosophy of Airflow, since this task would be affecting the host machine by building new docker images directly on it.
It's not the responsability of Airflow to apply this kind of use case.
Airflow is a pipeline and tasks orchestrator based on DAGs (direct acyclic graph).
Your need corresponds to usual CI CD pipelines.
It's better to delegate this work on a tool like Cloud Build or Gitlab CI for example.
From Cloud Build, you can apply and automate all the actions specified in your question.
When you will build your image in the CI CD part, you can then in the Airflow DAG, use a Docker image if needed with KubernetesPodOperator.
This would be more coherent because each concern will be put in the right place and on the right tool.
Related
I have docker images hosted external at docker hub which get updates every week.
Currently i did
Docker pull
Update some config files in the docker
Docker commit
Docker push
Then manually change the image name at kubernetes deployment yaml file.
What is the best practise for me to automate this? Can this be initiated in kubernetes?
K8s doesn't support such a functionality (yet, at least!)
but you can use GitOps tools like Flux to automate this procedure,
also you could use scheduled jobs of k8s combined to bash or python scripts to automate the task.
you better check out this post too:
Auto Update Container Image When New Build Released on Kubernetes
I am looking for any sample project to create MarkLogic 3 Nodes Cluster directly from ML Docker Hub image (https://hub.docker.com/_/marklogic) via Gradle on a single machine.
The idea is to auto spin off different ML versions for the dev enviroment set up.
Current three node cluster example in gitbub ml-gradle is to install ML from rpm installation package.
I would like to directly use the ML docker hub image instead.
MarkLogic on Docker Hub includes instructions for spinning up a cluster with this simple command:
docker-compose -f cluster.yml up -d --scale dnode=2
To run this, pull down the Docker image (you'll need an account on Docker Hub (free) and you'll need to do the checkout process to get access to the MarkLogic image (also free)). Then you can create the cluster.yml file using the example given on the setup instructions page on Docker Hub.
As #rjrudin points out, you can set up a gradle task to do this.
ml-gradle is typically used to deploy an application to an existing ML cluster. To actually create the ML cluster, you use the "docker" executable. You can automate this via Gradle's Exec task if you wish, but doing so is outside the scope of ml-gradle which assumes that you already have an ML cluster setup.
Our application has a Dockerfile that describes a custom image we'd like to use for to build and test application.
Basically, for every git push we want to:
Build an image from a Docker file.
Run a container based on this image.
Run build and tests in a container.
Get test results back to GitLab.
While it seems to be absolutely doable with GitLab CI's Shell Executor, I'm wondering if there is a recommended way to do such a thing?
Also, does this plan sounds appropriate for GitLab CI + Docker combination?
I'm creating an instance template under GCP's Cloud Engine section.
Usually when deploying a docker image, there's a docker file that includes some startup scripts after specifying the base image to pull and build, but I can't see where I can either submit a docker file, or enter startup scripts.
I can see a field for startup scripts for the Cloud Compute instance, but that's different from the scripts passed on for the Docker's startup.
Are these perhaps to be filled in under "Command", "Command arguments", or "Environment Variables"?
For clarification, this is someone else's image of a dockerfile I pulled from Google Images. The part I wish to add is "rectangled" in red, the RUN commands, but not these exact commands.
In my case, I would like to add something like
RUN python /pythonscript.py
If I understood well, you are trying to create a Docker image not a compute instance image.
Compute instance can run a docker image that you already builded and pushed to either gcr or any other repository.
Try to build your docker image normaly, push it in a docker repo then use it.
You can run a startup script directly in the Docker container by using a ‘command’ section. If you need to install something after starting a container for example Apache, you should use a Docker image that has Apache.
If you need to run some other arguments, like creating environment variables, here you can find the full list of flags when creating a container image on VM instance.
I currently run a Jenkins instance inside a Docker container. I've been doing some experimenting with Jenkins and their pipelines. I managed to make my Maven app build successful using a Jenkinsfile.
Right now I am trying to automatically deploy the app I built to a docker container that's a sibling to the Jenkins container. I did already mount the /var/run/docker.sock so I have access to the parent docker. But right now I can't seem to find any good information to guide me through the next part, which is modifying the Jenkinsfile to deploy my app in a new container.
How would I go over and run my Maven app inside a sibling docker container?
It might be more appropriate to spin up a slave node (a physical or virtual box) and then run something in there with docker.
If you check your instance URL: https://jenkins.address/job/myjob/pipeline-syntax/
You will find more information about what are the commands you need for this.
Anyway best way to do this is to actually create a dockerfile and as a step copy the artifact in it, push the image to a registry and then find somewhere to deploy the just built image