How can we limit Jenkins users to archive a large amount of files/large files in their jobs? It should be something on the Jenkins management level, but we didn't find the way to do so. Is there an option to limit a max size of archived file?
I don't know of any way built into Jenkins to do this, but haven't done an exhaustive search.
As a workaround you could try to use separate partitions/volumes or quotas for each jobs directory. I haven't done extensive testing, but Jenkins seems to continue to work after one job hits no space left.
This answer explains how to use groups and ACLs to make sure files in directory always get assigned that group. Then you can use group quotas to restrict the space usage for each.
This answer is an example of how to use project quotas which could here be an alternative to group quotas.
Related
I have a Jenkins instance that has users on different projects. I also have a Bitbucket instance where the users store their code. Push/pull access to the code repositories is authenticated by SSH public/private keys. Not all users in Bitbucket have access to all repositories.
I want users to be able to configure their jobs to use their own private keys to pull source from Bitbucket, but users should not be able to access other users' private keys. Is this possible with combination of the Jenkins Credentials plugin and the Git plugin? If so, how do I configure this?
If this is not possible with the Jenkins Credentials plugin/Git plugin, how can I implement this in Jenkins? I figure enterprises that use Jenkins must have resolved this problem but I can't find a solution anywhere.
Jenkins has a lot of inherent security issues in this respect and allowing users to have configuration permissions in a multi-tenant environment is very difficult to lock down.
Even if you were able to set permissions on a per credentials basis, a user that has configuration permissions to setup their own freestyle job can easily run processes on the nodes that could scrape passwords from the environment of other tenants. This can even be done with background processes if you limit one executor slot per node.
From what I've seen in industry and leveraged myself for an enterprise scale there are two high-level recommendations I would suggest:
Breakup the single instance into multiple instances so each set of
users or teams can have their own instance to work with.
Evaluate what users' functional needs are and provide a capability to request jobs where configuration permissions do not need to be given to users, rather only build/read permissions.
For Item 1:
Breaking up the instances not only helps Jenkins management from a security perspective, but from a scaling perspective as well since there are several issues you can run into with Jenkins once reaching a certain size (e.g. users that are resource hogs, unstashing bottlenecks, archiving bottlenecks, poorly written pipelines, etc.). These scaling problems typically lead to a need of vertically scaling the Jenkins master.
However, this approach has its own set of issues to solve since you now have multiple instances to maintain, but that is typically a bit easier to manage and there are some off-the-shelf solutions available if you're willing to pay the price (e.g. Cloudbees CI). Managing multiple instances can be solved in-house as well if you're willing to write some scripts or setup a service to handle this. Personally I'm a bit more of a fan of the in-house solution than the paid solution since I lean towards the ability to control one's own destiny and off-the-shelf solutions aren't always the one-size-fits-all they claim to be.
For Item 2:
If you really want to keep a single instance, the best way to secure it is to not let users have configuration permissions. As mentioned above, Jenkins has a lot of inherent security issues that does not make it well-suited for users to configure jobs in a multi-tenant setting. By evaluating the users' needs, you often find that there is a lot of common requirements users have that could be provided from common job templates without having to give them permissions.
Leveraging the Job DSL plugin to parameterize job creation is one way to do this. Parameters could then either be provided through a custom service or configuration files that are committed to a git repo. Another approach is leverage Jenkins REST API directly with a custom service that posts new job configurations from a customs job templates.
However, this approach could still run into scaling problem in the long term if the utilization of the Jenkins instance is expected to increase. These scaling problem are not insurmountable and can be mitigated with vertical scaling or offloading some stashing/archiving activity, but eventually at a certain point it might make sense to re-evaluate going with Item 1, or even a combination of Item 1 with Item 2.
Conclusion:
I know this is not likely the answer you were hoping for, but if security is a major concern, then a multi-tenant Jenkins instance that allow users' configuration permissions is not they way to go.
I'm getting the following error message when attempting to run a pipeline:
The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.
According to the docs, the limit is 10 MB. However; I would like to know how big the graph actually is, to make debugging it easier.
Is there any way to see the size of the graph?
As mentioned in the comment, use the --dataflow_job_file option. Note that there's no need to specify a gcs path, you can write it out locally. You can also pass the --dry_run option to avoid actually submitting the job.
Can I trace changes made in the global configuration of Jenkins to be able to retrieve the users who did the changes?
The Audit Trail Plugin tracks who is
creating/configuring/deleting jobs and views or delete/save-forever/start a build
If you are looking for something that tracks changes made under the Manage Jenkins section (ex. Configure System page or Add Plugins), I don't know of anything right off hand that tracks those change.
In our system, we severely limit who can access the Manage Jenkins section, or Job configuration, for that matter, so tracking changes is a less of an issue. Having as few administrators as possible might solve your issue. In our system, we do allow users to add certain types of jobs manually or perform certain administration / configuration tasks, but we do this by way of other jobs which control exactly how things are set up. In this way, we can easily track who added the job, ran the build, etc.
When submitting many jobs, I get an error similar to
Project <my project> has insufficient quota(s) to execute this workflow
Since this is a batch job, why is my job not held until resources are available?
Holding it until the resources are available isn't always the best solution -- that may never happen depending on your total quota, behavior of other workloads, etc.
But having an option to do so seems like it could be a useful feature -- will note your request.
We have many build pipelines running in Jenkins, each for separate projects and with multiple jobs (test, integration, deploy, quality, performance, release etc).
I am looking for some sort of radiator that will provide an aggregate single-page view of all pipelines, indicating if any have single jobs broken.
However, having looked around I can't find anything suitable. Has anyone seen anything similar at all? Would appreciate some pointers before attempting to build one...
As per #Gerolds suggestion above, the Categorized Jobs View plugin is able to aggregate a number of select jobs and therefore provide a single-page view on multiple pipelines (see attached).
Radiator? Click on the + right of your view's tab on your Dashboard. There's a view of type Radiator:
Shows status with high visibility, suitable for display in an extreme feedback system. You can choose which jobs are to be displayed in which view, in the same way as the list view.
And there's also the Categorized Jobs View:
Besides showing items in the list format, allows creation of categories of jobs