Jenkins (on a Kubernetes node) is complaining it requires a newer version of Jenkins to run some of my plug-ins.
SEVERE: Failed Loading plugin Matrix Authorization Strategy Plugin
v2.4.2 (matrix-auth) java.io.IOException: Matrix Authorization
Strategy Plugin v2.4.2 failed to load.
- You must update Jenkins from v2.121.2 to v2.138.3 or later to run this plugin.
The same log file also complains farther down that it can't read my config file... I'm hoping this is just because of the version issue above, but I'm including it here in case it is a sign of deeper issues:
SEVERE: Failed Loading global config
java.io.IOException: Unable to read /var/jenkins_home/config.xml
I'd either like to disable the plug-ins that are causing the issue so I can see the Jenkins UI and manage the plug-ins from there, or I'd like to update Jenkins in a way that DOES NOT DELETE MY USER DATA AND JOB CONFIG DATA.
So far, I tried disabling ALL the plug-ins by adding .disabled files to the Jenkins plug-ins folder. That got rid of most of the errors, but it still complained about the plug-in above. So I removed the .disabled file for that, and now it's complaining about Jenkins not being a new enough version again (the error above).
Note: this installation of Jenkins is using a persistent storage volume, mounted with EFS. So that will probably help alleviate some of the restrictions around upgrading Jenkins, if that's what we need to do.
Finally, whatever we do with the plug-ins and Jenkins version, I need to make sure the change is going to persist if Kubernetes re-starts the node in the future. Unfortunately, I am pretty unfamiliar with Kubernetes, and I haven't discovered yet where these changes need to be made. I'm guessing the file that controls the Kubernetes deployment configuration?
This project is using Helm, in case that matters. But again, I hardly know anything about Helm, so I don't know what files you might need to see to make this question solvable. Please comment so I know what to include here to help provide the needed information.
We faced the same problem with our cluster, and we have a basic explanation about that, but not sure about it (The following fix works)
That error come with the fact that you have installed Jenkins via Helm, and their plugins through the Jenkins UI. It works if you decide to never reboot the pod, but if one day, jenkins have to make his initialization again, you will face that error.
Jenkins try to load plugins from the JENKINS_PLUGINS_DIR, which is empty, so the pod die.
To fix the current error, you should specify your plugin in the master.installPLugins parameter.
If you followed a normal install, just go on your cluster and
helm get values jenkins_release_name
So you may have something like that:
master:
enableRawHtmlMarkupFormatter: true
installPlugins:
- kubernetes:1.16.0
- workflow-job:2.32
By default, some values are "embedded" by helm to be sure that jenkins works, see here for more details: Github Helm Charts Jenkins
So, just copy it in a file with the same syntax and add your plugins with their versions. After, you have just to use the helm upgrade command with your file on your release:
helm upgrade [RELEASE] [CHART] -f your_file.yaml
Good luck !
Related
Fore some reasons I have to use Jenkins 2.32 and I need to install some plugins there. Machine with it has no internet, so I only can upload plugin file to install it from file.
So, the problem is there any easy way to obtain required plugin for required Jenkins version with all it's dependencies?
p.s.
I can't update Jenkins - it's out of my power.
p.p.s.
I find only way to download old versions of plugin, but by this way I can't check dependencies and required jenkins version before loading.
I had such an environment before.
Warning: it's an annoying process.
Because there was no internet, we uploaded all plugins manually, i.e. looking at the plugin page (e.g. https://plugins.jenkins.io/git/) and then downloading from the archive (e.g. https://updates.jenkins.io/download/plugins/git) the .hpi file. As you have to use a relatively old version of Jenkins you may want to check the changelog of the plugin, if you have to use an older version.
In addition on each plugin page the dependencies are listed and you have to repeat the above steps for each dependency.
The only good thing is that usually Jenkins gives you hints, which dependencies are missing after you uploaded a plugin.
You can probably extract the information out of the plugin-versions.json in the Jenkins Update Center.
For more information about the layout of update center, see this document.
You may also find my previous response on jenkins failed to install plugins - docker image (with groovy scripts) helpful
I added the Marathon plugin to Jenkins through the Jenkins management Web UI. It showed up as a list of available plugins to install. I also downloaded the HPI and added the plugin manually. In both cases, the Marathon option doesn't show up in my pipeline config. I'm following the steps here: https://dcos.io/docs/1.7/usage/tutorials/jenkins/#building-a-docker-image-and-deploying-it-to-marathon
Use version mesosphere/jenkins:3.0.1-2.32.2 in combination with persisting your Jenkins data on NFS. Installation and updates of plugins works for me with this combination.
You should consider missing functionality in older versions of Jenkins in DC/OS as described in this thread: https://github.com/mesosphere/dcos-jenkins-service/issues/105
Do you use a NFS share to persist your Jenkins data? In my experience you can not use DC/OS Jenkins properly without persisting Jenkins' data.
I have a Jenkins server hosted, which has a master node and couple of other slave configurations. Last night, the job that triggers the matrix based build configuration failed. I did a restart and performed clean up jobs via Jenkins but none of those fixed the issue. The initial error that was logged was:
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
Following which I performed a reload configuration from disk,followed by a manual restart via <jenkins_job_url>/restart, which even worsened the build system. The master went offline due to unavailability of space in /tmp folder, which I fixed by cleaning up the space. Following which I observed that the original slave server configuration is no longer seen. I had slave-0 and slave-1 still there, but slave-2 was no longer present. Instead, it got replaced with slave-3 configuration. Now the slave 0's and 1's seems to be working fine. However, slave 3's build are failing due to Failed to mkdirs. Is there a way I could revert back to the original configuration from where I started, since the steps I performed seems to make sense initially, but I had no idea it had so many repercussions? Any help is appreciated.
UPDATE1: I guess I should have used some of the configuration backup plugins available in Jenkins, but is there some specific directory other than $JENKINS_HOME where these configurations gets stored?
You should always backup ${JENKINS_HOME} before doing major changes.
Even better is to have a job based on time trigger that will do this for you once in a while.
Other than that - only physically restoring the hard drive to a previous state will get back your old configs. Once a config is overwritten in Jenkins - it is gone. Except when you are using Job Config History plugin. Though keeping manually created backups is better in my opinion: where's the insurance that JobConfigHistory won't disappear along with the job configs? :)
Aside from that, the mentioned plugin tracks system config too.
As mentioned by #Zloj, there is no easy way to repair once the changes gets overwritten. I ended up fixing the issues by deleting the slaves that were not working, remapping the existing builds to the newer slaves that I created via Copy of the existing slaves that were working, reducing the number of builds(by removing the ones from the matrix that aren't required) and finally, taking a backup via https://wiki.jenkins-ci.org/display/JENKINS/thinBackup plugin and backing up the configuration at Stash :)
for windows just delete .jenkins folder in your home directory. This will revert you to the original settings.
We have been using SCM Sync Configuration plugin and that has saved our butt many times. It stores all job configuration including global config in bitbucket. But the latest plugin will say that its no longer maintained. but I was able to pull the source code from github and rebuild it ourselves.
one word of caution...don't use global variables for storing passwords and keys...this plugin will sync them all to github. Strictly use Jenkins Credentials.
I have a Jenkins Server (1.510) on Win 2008 with ~100 jobs.
After installing and then uninstalling the CloudBees (Plugin_1, Plugin_2) set of plugins + restart I have the following issues:
half of the Jobs are now missing.
many plugins are not functioning well, for example the green-balls plugin is not working and also the entry to launch the backup plugin is missing
many built in Jenkins buttons such the new "Credentials" is missing from the "Jenkins Configure" Menu.
Looking at the FS, i still see all the jobs.
I already tried :
Using the reload configuration
Reinstalling the plugins
Reinstalling Jenkins with the same version once again
Still the jobs are missing
Any idea how to solve it?
Thanks,
Doron
When job is loaded, many of the related Java classes get instantiated. If instantiation fails, usually because some plugin has been removed and that class is no longer available at all, then the job is hidden.
I suspect you have accidentally removed some other plugin too.
Note: Before actually doing anything, take full backup! Easiest is to backup entire Jenkins folder, where the jobs, configuration etc reside.
Easiest solution might be to just install Jenkins from scratch, install the plugins you do need (see below for troubleshooting if you're missing some), then copy the jobs subfolder to the new Jenkins. It might be best to do any configuration under Manage Jenkins by hand, but you can also just try copying the related XML config files.
If you are missing a plugin and can't figure out which one, you should look at jenkins.out.log and jenkins.err.log log files and search for exceptions happening after Jenkins is started. That may give you a clue on what plugin you are missing.
You can also try editing the job XML files to remove build steps you identified from the exceptions (remember to take backups first!), then restart Jenkins or select Reload configuration from disk from Manage Jenkins page.
If not solved, but you find relevant-looking exceptions or something else interesting, please update the question with details.
After I upgraded Jenkins, one of my jobs disappeared. I found out that although my job directory still existed, the config.xml file inside had somehow gone missing.
I restored this file from a backup, after updating all the plugins that needed updating, and reloaded the configuration, and the job reappeared in Jenkins.
I am on a redhat linux box. I recently updated Jenkins to version 1.509 only to find that after doing so it has "forgotten" two of my jobs/projects. The jobs can still be found on my Jenkins machine under /var/lib/jenkins/jobs, but they no longer show up in the Jenkins GUI. I attempted to re-create them based off the configuration file I have, but I am not confident I have totally re-created the functionality they had.
I also tried to copy the job and or rename it hoping that would get jenkins to see it, but no luck. I had tried cp -r /var/lib/jenkins/jobs/JOB1 /var/lib/jenkins/jobs/JOB2. I also restarted the service a number of times. Finally I updated all of my plugins on the off chance that was somehow related.
So my question is "How can I get Jenkins to notice these jobs?" or failing that "can I run these jobs from in the terminal?"
NOTE: I am not discouraging others from upgrading Jenkins. After I upgraded Jenkins did complain about a number of things which I didn't pay enough attention to which I believe got me into this mess in the first place.
If I were you, I would try the Jenkins CLI (from $JENKINS_URL/cli) and use the create-job command and feed the job configuration file to the cli's stdin.
If that does not help, I would inspect Jenkins log files (you are saving stdout and stderr of Jenkins somewhere, right?) for any errors or clues. If the job failed to load because of some tag that you can guess is provided by a plugin, try to remove that part from the config file.
If that does not help, I would upgrade Jenkins. I think there might be some fixes related to this in the LTS version changelog since 1.509.
And above all... if I were you, I would start making backups of the job configuration files.
I regularly back up the global config.xml, all the job config.xml files and all the plugins. Using these I can set up my Jenkins from scratch. And I do that to set up a test instance where I try any plugin or Jenkins core upgrade. If I see no problems after running a few of the trickiest builds, I know I can upgrade the production instance with much more confidence.