We have around 30 Jenkins installs across our organization, both Windows and Linux. They are all used for different tasks and by different teams (e.g. managing Azure, manipulating data, testing applications etc.)
I have been tasked with looking at whether we could bring these all into one 'Jenkins Farm' but as far as I can see such a thing doesn't exist? Ultimately 'we' want some control and to minimize the footprint of Jenkins. The articles I have found don't recommend using a single Master server (with multiple nodes) because of the following:
No role-based access for projects (affecting other teams code)
Plugins can affect all projects
Single point of failure as there is only one master server
Is it best to leave these on separate servers? Are there any other options?
I believe Role based access for projects is possible using
https://wiki.jenkins.io/display/JENKINS/Role+Strategy+Plugin
However, a single master isn't ideal as you pointed out due to 'Plugins can affect all projects'. Probably best to have separate jenkins master nodes but configure agents such that they can be shared across teams/projects.
Related
I have a Jenkins instance that has users on different projects. I also have a Bitbucket instance where the users store their code. Push/pull access to the code repositories is authenticated by SSH public/private keys. Not all users in Bitbucket have access to all repositories.
I want users to be able to configure their jobs to use their own private keys to pull source from Bitbucket, but users should not be able to access other users' private keys. Is this possible with combination of the Jenkins Credentials plugin and the Git plugin? If so, how do I configure this?
If this is not possible with the Jenkins Credentials plugin/Git plugin, how can I implement this in Jenkins? I figure enterprises that use Jenkins must have resolved this problem but I can't find a solution anywhere.
Jenkins has a lot of inherent security issues in this respect and allowing users to have configuration permissions in a multi-tenant environment is very difficult to lock down.
Even if you were able to set permissions on a per credentials basis, a user that has configuration permissions to setup their own freestyle job can easily run processes on the nodes that could scrape passwords from the environment of other tenants. This can even be done with background processes if you limit one executor slot per node.
From what I've seen in industry and leveraged myself for an enterprise scale there are two high-level recommendations I would suggest:
Breakup the single instance into multiple instances so each set of
users or teams can have their own instance to work with.
Evaluate what users' functional needs are and provide a capability to request jobs where configuration permissions do not need to be given to users, rather only build/read permissions.
For Item 1:
Breaking up the instances not only helps Jenkins management from a security perspective, but from a scaling perspective as well since there are several issues you can run into with Jenkins once reaching a certain size (e.g. users that are resource hogs, unstashing bottlenecks, archiving bottlenecks, poorly written pipelines, etc.). These scaling problems typically lead to a need of vertically scaling the Jenkins master.
However, this approach has its own set of issues to solve since you now have multiple instances to maintain, but that is typically a bit easier to manage and there are some off-the-shelf solutions available if you're willing to pay the price (e.g. Cloudbees CI). Managing multiple instances can be solved in-house as well if you're willing to write some scripts or setup a service to handle this. Personally I'm a bit more of a fan of the in-house solution than the paid solution since I lean towards the ability to control one's own destiny and off-the-shelf solutions aren't always the one-size-fits-all they claim to be.
For Item 2:
If you really want to keep a single instance, the best way to secure it is to not let users have configuration permissions. As mentioned above, Jenkins has a lot of inherent security issues that does not make it well-suited for users to configure jobs in a multi-tenant setting. By evaluating the users' needs, you often find that there is a lot of common requirements users have that could be provided from common job templates without having to give them permissions.
Leveraging the Job DSL plugin to parameterize job creation is one way to do this. Parameters could then either be provided through a custom service or configuration files that are committed to a git repo. Another approach is leverage Jenkins REST API directly with a custom service that posts new job configurations from a customs job templates.
However, this approach could still run into scaling problem in the long term if the utilization of the Jenkins instance is expected to increase. These scaling problem are not insurmountable and can be mitigated with vertical scaling or offloading some stashing/archiving activity, but eventually at a certain point it might make sense to re-evaluate going with Item 1, or even a combination of Item 1 with Item 2.
Conclusion:
I know this is not likely the answer you were hoping for, but if security is a major concern, then a multi-tenant Jenkins instance that allow users' configuration permissions is not they way to go.
I am deploying a web app onto multiple servers. I would like to push code to one of these servers so that it can be updated but not affect the other deployed servers. The web app and server is unique to each client so if I add a feature for one client I would not like to add it to all the other servers that are deployed. There will be occasions where an update might be required for all the active servers.
I am hoping to automate this functionality so that I can push code for one server without it affecting the other servers. I have been looking into using Docker, Ansible, and Kubernetes for this job but am fairly new to deployment in general and would like to get an idea of the best practices for something like this.
Thank you.
It is now deployed as a stand alone web server where a push to the master branch causes an automatic deployment for that one server. The only solution I've come up with is deploying multiple servers individually.
Your question is not strictly related to Docker/Kubernetes but I will try to answer anyway. There are two ways this could be done
1) Maintain a repo for each customer - which has the customer's customization and pulls the common code from somewhere else. Maybe a Git submodule or some other way. This way each customer can have their own lifecycle. I don't like this approach because the drift between customer tends to get bigger. Also, it is like maintaining as many codebases as there are customers - which is not scalable
2) The only way I can think of this can be done in a way which is uniform for all customers is to do feature flags. This has to be during development and is not a purely operational exercise. This is difficult to get started but in longer-term is the only way I think this can be scaled. Specific features for the one customer then can be turned off by configuration. There are also some companies like LaunchDarkly which enable this
I've been looking all over the internet for a free way to load Jenkins up as an enterprise application. To be more clear, I mean to load Jenkins front end into two or more servers and allow load balancing between them.
Everything I've read is regarding distributed build. While I will also want to do this, make all servers build agents as well, I would like a disaster recovery environment kind of set up for the front end in the event that, say, our connection is down to a data center. Active/Active hosting would be desired, but active/backup would be fine too.
Any materials available to explain how to do this?
I haven't used this solution but it looks like the Gearman Plugin might provide the architecture you're looking for. It looks like the plugin creates a job queue that can be accessed by multiple Jenkins masters and serviced by multiple build agents. Looks like this would support an active/active set up.
https://wiki.jenkins.io/display/JENKINS/Gearman+Plugin
What are the best practices for a jenkins installation like the one below?
I do have a quite small dedicated server with 16 gb of ram and 2tb of diskspace with enabled hardware virtualization, and one use of it would be to host my own projects (opensource), and there are applications set up such as git repository manager and stuff.
I would like to set up jenkins there for automatic building, but I want to make it secure.
This installation is small enough to require only a master node, but I am planning to disable building on master completely, and to run a virtual machine as an agent, for the reason that it would be isolated as much as possible on the same physical server, so that a job would be unable to destroy jenkins master data.
Should I go for master only anyway? or, if using a virtual machine agent, should I have only one executor there or multiple ones? I probably cannot isolate multiple parallel jobs running on one agent without using one agent per job, but maybe I am overthinking all this. Using one agent per job, at least in case of virtual machines, would exhaust server resources very quickly, or alternatively, money.
You can use Jenkins own database of users, which I have used in commercial settings and it has worked perfectly well. If you have Active Directory you can also integrate with this if you want to go to extra effort so people only have to remember one login.
Once users are logged in you should provide authorisation via the Role-Strategy plugin
My company's CI uses Jenkins at the core, configured to run about 100 jobs -- one job per one Visual Studio solution.
For some reason (I have yet to determine why), my manager is eager to move to a job-per-project (Visual Studio .csproj, that is) Jenkins configuration. This is essentially going to increase the number of jobs to thousands.
My immediate thought was: Even if Jenkins is capable of handling thousands of jobs, what would be the user experience of the folks looking at the Jenkins dashboard like?
Some mitigation could come in the form of using tabs (views) to make it easier on the eye and on navigation (e.g. view/tab per solution or per some other class). But still... 100 tabs?
My questions are:
What is your experience with Jenkins managing/running thousands of jobs?
How did you solve the UX problem?
Are there other issues? (I am assuming performance is not an issue because in principle I can employ slave machines if I run out of resources).
What are the main benefits of running a job-per-project (vs. job-per-solution)
Partial answers/insights are very much appreciated as well.