Jenkins service not starting anymore after system update - jenkins

I have updated my redhat server where jenkins was installed. After reboot, the jenkins service is not starting up.
getting the below error
java.io.FileNotFoundException:
"/opt/data/jenkins/war/"META-INF/MANIFEST.MF (No such file or
directory)
MANIFEST.MF file is present under /opt/data/jenkins/war/META-INF . Any idea what this could be? Also the " in /opt/data/jenkins/war/ looks a bit strange to me.

We fixed this by looking at systemctl parameters which were wrong. Restart jenkins after correcting that solved the problem

Related

Jenkins System configuration error when saving

An instance of Jenkins started not saving changes made under 'Manage Jenkins > System Configuration'.
In an attempt to solve it, I have recently upgraded to Jenkins 2.346.3 (including all the plugins).
Unfortunately, this behavior still persists and the System Log only shows:
Error while serving http://<jenkins_url>/configSubmit
java.lang.ClassCastException: java.lang.Integer cannot be cast to hudson.model.Describable
at hudson.util.DescribableList.get(DescribableList.java:128)
at hudson.util.DescribableList.rebuild(DescribableList.java:170)
at jenkins.model.GlobalNodePropertiesConfiguration.configure(GlobalNodePropertiesConfiguration.java:24)
at jenkins.model.Jenkins.configureDescriptor(Jenkins.java:4017)
at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3981)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
Caused: java.lang.reflect.InvocationTargetException
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:401)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
<snippet>
Any idea on the possible cause?
UPDATE
After 2 attempts on restarting Jenkins without the config.xml, I succeeded in having Jenkins 'Manage Jenkins > System Configuration' behaving as expected.
After the first attempt, I reverted to the old configuration file as all the security related configurations were missing and I ended up raising the ticket https://issues.jenkins.io/browse/JENKINS-69548
On the 2nd attempt, I did what I described in the ticket comment https://issues.jenkins.io/browse/JENKINS-69548?focusedCommentId=430091&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-430091 (i.e. re-introducing the security-related configurations)
You probably have a corrupted config.xml from your old installation. Try deleting the config.xml(Back it up) located at $JENKINS_HOME(if you have not changed the default JENKINS_HOME in most cases it will be at USER_HOME/.jenkins(~/.jenkins)) and restarting Jenkins. If it's successful you can start reconfiguring or moving the configs from there.

How to fix Jenkins java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded

I upgraded from jenkins 2.219 to 2.272 (latest version as of this writing) and now getting the stack trace below when Jenkins starts.
Jenkins docs says that this happens due to the Configuration as Code plugin and to set the jvm args as -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000. The docs also say to increment the value until the error goes away but so far I'm at 480000 and still getting the error. I also don't see that I have the Configuration as Code plugin installed anyway.
How can this be fixed?
java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded
at jenkins.model.Jenkins.save(Jenkins.java:3379)
at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3398)
at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2637)
at jenkins.model.Jenkins$16.run(Jenkins.java:3342)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1162)
at jenkins.model.Jenkins.<init>(Jenkins.java:962)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:295)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:312)
For anyone having this issue, yes downgrading to https://get.jenkins.io/war-stable/2.263.1/ first is required. Then restart the service, then upgrade all of your plugins, then try the LTS upgrade again, at least for me it worked just fine.
More details about the issue here:
https://www.jenkins.io/doc/upgrade-guide/2.204/
SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed
ConfigurationAsCode.init java.lang.IllegalStateException: An attempt
to save the global configuration was made before it was loaded If
you encounter this, you can tell the plugin to delay configuration for
an amount of time to give Jenkins time to load the global
configuration before the configuration is applied by the plugin.
To enable this set the
io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system
property to a number of milliseconds to delay the initialisation. The
required value will be dependant on aspects of your system (cpu/disk)
and configuration, and how it can be found is mostly a trial and
error. A suggestion would be to start with 5000 (5 Seconds) and then
increment by 2000 (2 seconds) until you no longer exhibit the issue
and finally add 1000 (1 second) for some extra safety. For example, to
delay the configuration by 9 seconds you would use something like the
following command java
-Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000 -jar jenkins.war. Exactly how and where you specify this option depends on
the installation method used to install Jenkins
However for me the workaround didnt work without first downgrading, upgrading all plugins, then finally upgrading the core again.
To avoid this issue beforehand when you actually have not installed the Casc plugin:
Update to https://github.com/jenkinsci/role-strategy-plugin/releases/tag/role-strategy-3.1 and you should be OK.
When jenkins is already in the faulty state:
download the hpi and drop the hpi file into JENKINS_HOME/plugins folder.
https://updates.jenkins.io/download/plugins/role-strategy/
(information taken from: https://github.com/jenkinsci/configuration-as-code-plugin/issues/1531)
I faced the same issue. I have downloaded the jenkins.war , the earlier version (2.263.1) from https://www.jenkins.io/download/ . Stopped the jenkins and replaced the war with my install directory(C:\Program Files\Jenkins). And started the jenkins.This works like a charm.
You can download the hpi and drop the hpi file into JENKINS_HOME/plugins folder. Usually the jenkins plugin directory will be under - /var/lib/jenkins/plugins/.
The link to get Role strategy hpi plugin - https://updates.jenkins.io/latest/role-strategy.hpi. And restart Jenkins using this command: systemctl restart jenkins.
My scenario was to upgrade the Jenkins to the latest version and then restore the data in it to work like it was working before.
While upgrading from Jenkins v2.263 to v2.303, I had the same error. The only thing which worked for me was to upgrade my global plugin i.e Role-based authorization strategy first, and then Jenkins. It worked like a crisp.
We had the same issue after we upgraded to the latest version.
After some searching we decided to go with a lower version which has LTS support, 2.263.1 in this case.
That version worked instantly so I think this error is a bug.
we has similar issue, resolved by downgrading the jenkins.war file downloaded from URL: https://get.jenkins.io/war-stable/2.263.1/
I have faced similar issue, and found that we don't need to downgrade version.
Below are the steps I performed and it worked for me
upgrade global plugin i.e Role-based authorization strategy first from Jenkins Console.
Here are steps which I have performed:
#on ubuntu, in /usr/share/jenkins:
sudo service jenkins stop
sudo mv jenkins.war jenkins.war.old
sudo wget https://updates.jenkins-ci.org/latest/jenkins.war
sudo chown -R jenkins:jenkins jenkins.war
sudo service jenkins start
I have the same error on Jenkins 2.332.2 and updated plugin role-strategy v483.v17281966f5c3.
I had to download the version 3.2.0 of the plugin and put it directly in the jenkins plugins folder. (previously I remove the existing '.jpi' file)
Then restarting Jenkins /etc/init.d/jenkins restart, and the issue has gone.
Thank you for the intel!

Jenkins Master: Updating plugin data fails

On my Jenkins Master, versio 2.203, when I'm trying to update the plugins, I have this error:
java.security.cert.CertificateException: No subject alternative DNS name matching updates.jenkins.io found.
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:207)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:98)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:429)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625)
I have tried the option JENKINS_JAVA_OPTIONS ="(..) -Djsse.enableSNIExtension=false" but didn't work.
Any idea what I can do?
Thank you.
Removing -Djsse.enableSNIExtension=false from the Jenkins Java options fixed it for me. On Ubuntu, I went to the etc/default/jenkins file, located JAVA_ARGS and removed -Djsse.enableSNIExtension=false from this line. You need to reboot the mahcine after that for the new settings to get in effect before you try to update the plugins again on Jenkins.

I can't change java memory for Jenkins

Note: My OS Debian 7, Java 8, 2Gb RAM
Here is what I did:
I went this way: /etc/default/
Then: vim jenkins
Then edit: JAVA_ARGS="-Xmx1048m"
I exit from editor with saving: :wq
Now I make do: service jenkins restart
Now I go to my Jenkins and watching monitoring with plugin JavaMelody.
So, I see that no changes have occurred.
I ask for help in this case, please.
From the official guide, if you're using RedHat Linux based distributions, you should use JENKINS_JAVA_OPTIONS.
JENKINS_JAVA_OPTIONS="-Xmx1048m"
Note: For me, the file location was /etc/sysconfig/jenkins and it only had JENKINS_ARGS="". Assigning Xmx value to it did not work. You should leave that entry as is and instead, add the JENKINS_JAVA_OPTIONS entry in the file just like i specified above.
The problem is resolved and I did the following:
I went this way: /etc/default/
Then: vim jenkins
I'm add in jenkins JAVA=/usr/bin/java
and JAVA_ARGS="-Djava.awt.headless=true -Xmx1024m"
I exit from editor with saving: :wq
Then after I made: service jenkins restart
Now I go to my Jenkins and watching monitoring with plugin JavaMelody.
So, I see that this works for me.
I'm hope this helps someone.

Jenkins job lost upon restart even though configuration is on the disk

I have restarted Jenkins using the following:
service jenkins stop
service jenkins start
Followed to that I can see some jobs are missing from the GUI.
I have also tried to go the job URL using http://<jenkins_url>/job/<JOBNAME>/
Unfortunately it is also giving:
HTTP ERROR 404
Problem accessing /job/<JOBNAME>/. Reason:
Not Found
Powered by Jetty://
Also performed Doing a Reload Configuration from Disk with no luck.
I checked the config.xml file and I can see it is corrupted. The size of config.xml file is around 110 MB. Why this file got corrupted? How to trace it.
Can anyone give me any pointer how to troubleshoot this problem?
I had the same symptoms, but I'm using a homebrew installation of Jenkins.
The Jenkins machine was shut down improperly, likely from a power outage, so when it came back up it was basically a clean instance. No jobs and no system configurations.
The following solution isn't for your exact use case, but it does solve the problem for some users who return to Jenkins to find it without any jobs.
The solution basically involves you checking to see if you have started the Jenkins service incorrectly or from the wrong place.
...
On to the specific homebrew issue:
For whatever reason, the homebrew.mxcl.jenkins.plist file was found in ~/Library/LaunchDaemons/
It belongs in ~/Library/LaunchAgents/ only.
If this happens, it can be solved as follows
Stop the service:
sudo launchctl unload ~/Library/LaunchDaemons/homebrew.mxcl.jenkins.plist
Reload the correct file, located in ~/Library/LaunchAgents/ by trying the following line in case it's running:
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.jenkins.plist
Note: the above line may yell at you if it's not running, which is ok.
Start it up again:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.jenkins.plist
If all looks good when Jenkins loads again, you can and should
delete homebrew.mxcl.jenkins.plist in ~/Library/LaunchDaemons/:
sudo rm ~/Library/LaunchDaemons/homebrew.mxcl.jenkins.plist

Resources