Our team is developing Kafka Connect Source Connector-plugins.
Do you have any ideas on how to install/upgrade the plugins? How is the flow (git -> Jenkins -> running Source Connector) supposed to look on-prem?
We use Confluent on Kubernetes which complicates things further.
PS. We are required by law to not use cloud solutions.
To store custom connectors, use Nexus, Artifactory, S3, or some plain HTTP/file server.
If you are using Kubernetes, then you probably have a release policy around your Docker images.
Therefore, you can extend the Confluent Connect Docker images by adding additional RUN statements to the Dockerfile, then build and tag your images with Jenkins, and upgrade your Kubernetes services to use the new image tag.
The answer I would give for a bare-metal (or cloud) installation of managing Kafka Connect would be to use Ansible or other orchestration tool to push out the new files, and restart the services
Related
I have a docker container which I push to GCR like gcloud builds submit --tag gcr.io/<project-id>/<name>, and when I deploy it on GCE instance, every time I deploy it creates a new instance and I have to remove the old instance manually. The question is, is there a way to deploy containers and force the GCE instances to fetch new containers? I need exactly GCE, not Google Cloud Run or other because it is not an HTTP service.
I deploy the container from Google Console using the Deploy to Cloud Run button
I'm posting this Community Wiki for better visibility. In the comment section there were already a few good solutions, however at the end OP wants to use Cloud Run.
At first I'd like to clarify a few things.
I have a docker container which I push to GCR like gcloud builds submit
gcloud builds submit is a command to build using Google Cloud Build.
Cloud Build is a service that executes your builds on Google Cloud Platform infrastructure. Cloud Build can import source code from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives.
In this question, OP is referring to Container Registry, however GCP recommends to use Artifact Registry which soon will replace Container Registry.
Pushing and pulling images from Artifact Registry is explained in Pushing and pulling images documentation. It can be done by docker push or docker pull command, where earlier you have to tag an image and create Artifact Registry.
Deploying on different GCP products
Regarding deploying on GCE, GKE and Cloud Run, those are GCP products which are quite different from each.
GCE is IaaS where you are specifying the amount of resources and you are maintaining all the installation of all software (you would need to install Docker, Kubernetes, programming libs, etc).
GKE is like Hybrid as you mention the amount of resources you need but it's customized to run containers on it. After creation you already have docker, kubernetes and other software needed to run containers on it.
Cloud Run is a serverless GCP product, where you don't need to calculate the amount of needed resources, installing software/libs, it's a fully managed serverless platform.
When you want to deploy a container app from Artifact Registry / Container Registry, you are creating another VM (GCE and GKE) or new service (Cloud Run).
If you would like to deploy new app on the same VM:
On GCE, you would need to pull an image and deploy it on that VM using Docker or Kubernetes (Kubeadm).
On GKE you would need to deploy a new deployment using command like
kubectl create deployment test --image=<location>-docker.pkg.dev/<projectname>/<artifactRegistryName>/<imageName>
and delete the old one.
In Cloud Run you can deploy an app without concerns about resources or hardware, which steps are described here. You can create revisions for specific changes in the image. However Cloud Run also allows CI/CD using GitHub, BitBucket or Cloud Source Repositories. This process is also well described in GCP documentation - Continuous deployment
Possible solutions:
Write a Cloudbuild.yaml file that do that for you at each CI/CD pipeline run
Write a small application on GCE that subscribes to Pub/Sub notifications created by Cloud Build. You can then either pull the new container or launch a new instance.
Use Cloud Run with CI/CD.
Based on one of the OP's comments, as chosen solution was to use Cloud Run with CI/CD.
I am not able to integrate Docker with Jenkins completely. I setup Docker at Cloud Node settings in the Jenkins portal. How can I use some image that contains Node, Ruby, etc? Must those images contain a JDK?
There is a great post with a detailed explanation of how to deal with this here. In the post, they state that every Docker image (even if you require building a Node or Ruby project) requires a JDK. Seems the JDK requirement comes from the Docker plugin.
I have a monolithic application that I am hosting on google cloud.
I am using cloud build that builds my docker image when I push to my repository.
Other than using Kubernetes, what other options do I have to push my latest docker image to my web instances in a rolling update to not bring my website down?
I cant' seem to find any documentation other than Kubernetes related.
I believe I should be building a instance template that has my latest docker image. Not sure how to make this happen in an automated fashion.
We have a requirement to build custom docker images from base docker images with some additional packages/customization. These custom docker images need to be then deployment into kubernetes. We are exploring various tools to figure out on how docker build can be done in kubernetes cluster (without direct access to docker daemon). Open source tools like kaniko provides the capability to build docker images within a container (hence in a kubernetes cluster).
Is it a good practice is build docker images in kubernetes cluster where other containers will be run/executed? Are there any obvious challenges with kaniko?
Should separate dedicated VMs be created to manage the build process?
1. Is it a good practice is build docker images in kubernetes cluster where other containers will be run/executed?
Are there any obvious challenges with kaniko?
Yes, it is possible to build images inside Kubernetes containers, but it could be a bit of a challenge.
Some users use it to build a workflow for CI/CD with Jenkins. In fact, it is better to use tools to simplify the process.
Kubernetes also have rules to prepare containers development kit, they are described here
Another way is to use Kaniko, this tool builds container images from a Dockerfile inside a container or Kubernetes cluster.
I found this article interesting to read on this topic.
On the other hand, there was a successful attempt to build images without Docker daemon running. You may be interested in Bazel project and story how to use it.
2. Should separate dedicated VMs be created to manage the build process?
Regarding your second question: It is not necessary to set up dedicated VM to run Docker images creation workflow.
Finally, it may be interesting to have a private registry in Kubernetes cluster and use it for building purposes.
It's possible to build images on kubernetes nodes. But i wouldn't recommend it. The reason being, a application build process is memory and compute intensive, frequent image builds could cause disruption to services being scheduled by that kubernetes node.
Use a dedicated Jenkins server(s) instead, create pipelines according to your requirements and delivery.
You can get started here!
Hope that helps!
I read this How does docker compare to openshift?
But I have a question :
This is an extremely simplified description of what usually devs do with Openshift :
Select a "pod" (let's say a JBoss/Wildfly container)
From within Openshift you point to your github repo
Openshift would clone the repo, build it and deploy it
Openshift present you with a web URL to access this repo port 8080
There's of course a lot more going on but that's as simple as it gets
Is this setup doable in my own linux box, VM or a cloud instance (Docker Container --> clone, build and deploy from git repo)? What would I need without messing too much with networking and domains etc?
from my research I see the following tools:
Kubernetes
Dokku : I see it described as "Your own Heroko"
I also keep hearing about CaaS (Containers as a Service)
I understand I would be needing another tool or process to the build (CI/CD) capability, and to triggering builds with git push.