We are considering moving Jenkins to Kubernetes (GKE) and using Jenkins Kubernetes plugin to run job in pods. However, our build infra uses ccache for build speed optimization and we keep ccache files on the jenkins agents (every build reads and writes to local ccache folder on each agent).
Per my understanding Jenkins agent pods in k8s are ephemeral, meaning they are for a single use only. I am struggling to understand how ccache set up in going to work. Here is what I am thinking:
Use hostpath volume in the pod templates to have agent pod mount a path on a node. I am afraid that is not going to work because based on GKE documentation hostpath doesn't allow ReadWriteMany, meaning multiple pods won't be able to write to ccache mount at the same time.
Use NFS based volume - I know for a fact that won't work well as it's going to be slow and will have locking problems.
Somehow configure pod templates with 'podRetention' and 'idleMinutes' option to have Pods to be re-used. I think this is doable, but IMO this defeats the purpose of using Kubernetes.
Use StateFull Jenkins Agents - again I think this is doable but defeats the purpose of using Kubernetes.
I found https://issues.jenkins.io/browse/JENKINS-42422 which might be related, but I don't think there is a solution.
Any other ideas on how this can be implemented? Not being able to use ccache is a deal breaker for us as it will make build turnaround time way worse compared to static agents.
Being stateful does not defeat the purpose of Kubernetes. Stateful apps, are necessary, for example, if you are running a DB in your cluster.
I would suggest that you use glusterfs/ceph/nfs/EFS(AWS) and use readWriteMany for ccache.
This blog has more details:
https://blog.hiya.com/kubernetes-base-jenkins-stateful-agents/
Related
I am trying to setup Kubernetes for my company. I have looked a good amount into Jenkins X and, while I really like the roadmap, I have come the realization that it is likely not mature enough for my company to use at this time. (UI in preview, flaky command line, random IP address needs and poor windows support are a few of the issues that have lead me to that conclusion.)
But I understand that the normal Jenkins is very mature and can run on Kubernetes. I also understand that it can have dynamically created build agents run in the cluster.
But I am not sure about gitops support. When I try to google it (gitops jenkins) I get a bunch of information that includes Jenkins X.
Is there an easy(ish) way for normal Jenkins to use GitOps? If so, how?
Update:
By GitOps, I mean something similar to what Jenkins X supports. (Meaning changes to the cluster stored in a Git repository. And merging causes a deployment.)
I mean something similar to what Jenkins X supports. (Meaning changes to the cluster stored in a Git repository. And merging causes a deployment.)
Yes, this is the what Jenkins (or other CICD tools) do. You can declare a deployment pipeline in a Jenkinsfile that is triggered on merge (commit to master) and have other steps for other branches (if you want).
I recommend to deploy with kubectl using kustomize and store the config files in your Git repository. You parameterize different environments e.g. staging and production with overlays. You may e.g. deploy with only 2 replicas in staging but with 6 replicas and more memory resources in production.
Using Jenkins for this, I would create a docker agent image with kubectl, so your steps can use the kubectl command line tool.
Jenkins on Kubernetes
But I understand that the normal Jenkins is very mature and can run on Kubernetes. I also understand that it can have dynamically created build agents run in the cluster.
I have not had the best experience with this. It may work - or it may not work so well. I currently host Jenkins outside the Kubernetes cluster. I think that Jenkins X together with Tekton may be an upcoming promising solution for this, but I have not tried that setup.
We have a requirement to build custom docker images from base docker images with some additional packages/customization. These custom docker images need to be then deployment into kubernetes. We are exploring various tools to figure out on how docker build can be done in kubernetes cluster (without direct access to docker daemon). Open source tools like kaniko provides the capability to build docker images within a container (hence in a kubernetes cluster).
Is it a good practice is build docker images in kubernetes cluster where other containers will be run/executed? Are there any obvious challenges with kaniko?
Should separate dedicated VMs be created to manage the build process?
1. Is it a good practice is build docker images in kubernetes cluster where other containers will be run/executed?
Are there any obvious challenges with kaniko?
Yes, it is possible to build images inside Kubernetes containers, but it could be a bit of a challenge.
Some users use it to build a workflow for CI/CD with Jenkins. In fact, it is better to use tools to simplify the process.
Kubernetes also have rules to prepare containers development kit, they are described here
Another way is to use Kaniko, this tool builds container images from a Dockerfile inside a container or Kubernetes cluster.
I found this article interesting to read on this topic.
On the other hand, there was a successful attempt to build images without Docker daemon running. You may be interested in Bazel project and story how to use it.
2. Should separate dedicated VMs be created to manage the build process?
Regarding your second question: It is not necessary to set up dedicated VM to run Docker images creation workflow.
Finally, it may be interesting to have a private registry in Kubernetes cluster and use it for building purposes.
It's possible to build images on kubernetes nodes. But i wouldn't recommend it. The reason being, a application build process is memory and compute intensive, frequent image builds could cause disruption to services being scheduled by that kubernetes node.
Use a dedicated Jenkins server(s) instead, create pipelines according to your requirements and delivery.
You can get started here!
Hope that helps!
I wanted to build a Jenkins server which would run test of my puppet code on Vagrant. The issue I found is that the we run our server as VMs already, either in vmWare or AWS and Vagrant will not work as another virtualisation.
Does anyone have an idea how can I create a test platform for my puppet code. What I want to test the deployment of manifest on the nodes them self i.e. If I deploy a class web server or make changes to it I would like to check if it affects/breaks deployment of other classes.
The idea would be to iterate over all the classes/roles and see if the deployments are passing. I would like to make it automatic and independent of our engineers. At the moment we are running manual test with vagrant up however there are too many roles to do that by hand.
Any ideas how can I tackle this?
You can use either Docker or AWS provider for Vagrant.
In case of AWS provider you need to set-up RSync to get your environment into newly launched instance.
If your Vagrant scripts are robust, you can use the same script for both local deployment on your workstation and AWS/Docker deployment on CI server.
There are drawbacks to doing these techniques, in case of Docker you are limited to the same kernel that Jenkins server is running, in case of AWS you will incur additional costs. However, for AWS your don't need to allocate as much resources for your Jenkins server, so you might even save money this way because you will be using paying for extra VMs only when you are running you tests. Just make sure you will shut them down after you done.
Is there any special reason why you want to use vagrant? I'm not sure if you are setting up your production environment with vagrant or not.
In case you are not bound to vagrant, I would recommend you to think about using a docker image to prepare a lightweight environment to run your setups and verifications in.
When doing your tests, spin up a container from your image that contains your puppet distribution and run your setups/tests inside. If you have special kernel requirements, use a separate jenkins slave/agent machine rather than executing jobs on the jenkins master.
If you are not sure how to get started using jenkins with docker, have a look into the examples section of the Jenkins Documentation. The provided examples are showing the declarative pipeline syntax thats still a bit new. Also consider the collapsed Toggle Scripted Pipeline Sections which show the groovy pipeline scripts that are alot more forgiving for jenkins pipeline beginners.
Those should be quite good pointers to get started with running+testing your puppet scripts inside docker. For building and using a docker image there should be more than enough tutorials out there.
Let me know if this was a hint in the right direction or if I mistinterpreted your question.
Suppose i want to move mu current acceptance test CI environment to dockers, so i can take benefit of performance improvements and also quickly setting up multiple clones for slow acceptance tests.
I would have a lot of services.
The easy ones would be postgres, mongodb, reddis and such, which are updated rarely.
However, how would i go about, if my own product has lots of services aswell? - over 10-20 services, that all need to work together for tests. Is it even feasible to handle this with dockers, i.e., how can CI efficiently control so many containers automatically AND make clones of them to run acceptance tests in parallel.
Also, how would i automatically update the containers easily for the CI? Would the CI simply need to rebuild every container at the start of the every run with the HEAD of every service branch? Or would the CI run git pull and some update/migrate command on every service?
In VM-s its easy to control these services, but i would like to be convinced that dockers are good or better for it as well.
I'm in the same position as you and have recently gotten this all working to my liking.
First of all, while docker is generally intended to run a single process, for testing I've found it works better for the docker container to run all services needed. There is some duplication in going this route, but you don't have to worry about shared services, like Mongo or PostgreSQL. This can be accomplished by using something like Supervisor: http://docs.docker.com/articles/using_supervisord/
The idea is to configure supervisor to start all necessary services inside the container, so they are completely isolated from other containers. In my environment, I have mongo, xvfb, chrome and firefox all running in a single container. So really, you still are running a single process (supervisor) but it starts many others.
As for adding repositories to your container, I just have the host machine checkout the code and then when I run docker, I use the -v flag to add the repo to the container. This way you don't need to rebuild the container each time. I build containers nightly with the latest code to be able to add all necessary gems for a faster 'gem install' at testing time.
Lastly I have a script as the entrypoint of the container that allows me to pass in what test I want to run.
Jenkins then just runs the docker commands and passes in the tests to run. These can be done in parallel, sequentially or any other way you like. I'm currently looking into having these tests run on slave Jenkins instances in an auto-scaling group in AWS.
Hope that helps.
drone is a docker based open source CI plus online service: https://drone.io
Generally it runs build and test in docker containers, and remove all containers after built. you just need to provide a file named .drone.yml with similar configuration like .travis.yml to configure your build.
it will manage your services like database, cache as linked container.
For your build environment, you can use exiting docker images as template of dependencies.
So far, it supports github.com and gitlab. for your own CI system, you can use drone CLI only or its web interface.
I recommend to use Jenkins docker plugin, though it is new, it starts to expose the power of docker used inside jenkins, the configuration is well written there. (let me know if u have problem)
The strategy I planned to use it.
create different app images to serve different service like postgres, mongodb, reddis and such, since it is rare updated, they will be configured globally as "cloud" template in advance, each VM will have label to indicate the service
In each jenkins job, each images will be selected as slave node (use that label as name)
When the job is triggered, it will automatically start the docker container as slave in seconds
It shall work for you.
BTW: As the time I answered (2014.5), the plugin is not mature enough, but it is the right direction.
I want to deploy a private cloud test infrastructure using OpenStack and Jenkins for multiple projects. I thought of creating a template for OpenStack with one Jenkins installation using as master. For the projects I thought of separating them into nodes, i.e. each project would get one node. Is this a sensible structure? Or should I install one Jenkins installtion per project+vm?
1) How would you organize a private multi-project test cloud infrastructure?
2) Jenkins saves configuration and job information to /var/lib/jenkins by default, how do I manage the object storage for each project?
When you say node, I'm assuming you mean a machine running nova-compute and hosting VM instances. If this is the case, then I honestly wouldn't worry about trying to bind a project to a specific node - treat the entire openstack pool of resources you have as a global cluster, assign in projects, and let them spin up and tear down as they need.
You will likely find it beneficial to have an image with jenkins pre-installed as a publically available image, assuming you want a master jenkins per project in your cloud. If you're running jenkins as a stand-alone item per project, using a m1.medium may be sufficient, but you might find you want to use m1.large. It all depends on what you have your jenkins instance doing in each project.
If you want the jenkins data to persist across destroying and recreating the jenkins master instance, then you could use a volume and specifically mount /var/lib/jenkins into it - but you will need to manage the coordination of jenkins startup and having the volume attached appropriately. You may find it easier to give the jenkins instance a larger base disk and just back up and restore the data per project if you need to destroy and recreate the jenkins instance.
Another way to do this would be to share a master jenkins external to your openstack cloud and use the jclouds jenkins plugin to spin up jenkins instances and slaves as you need for projects. This isn't providing any segregation between projects in jenkins, which may not be to your liking based on the question above.