During my Jenkins maintenance time, I tried to "Update" some Plugins & "Uninstall" \"Unpin" some plugins at the same attempt, from "Jenkins -> Plugin Manager -> Updates \ Installed" Tabs and restarted my Jenkins Instance.
Jenkins failed to Load\parse my Config.xml saying below Errors in diff Attempts.
WARNING jenkins.diagnosis.HsErrPidList
Failed to list up hs_err_pid files
java.lang.UnsatisfiedLinkError: com.sun.jna.Native.open(Ljava/lang/String;)J
at com.sun.jna.Native.open(Native Method)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:236)
at com.sun.jna.Library$Handler.<init>(Library.java:140)
Unable to read c:\jenkins\config.xml
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:246)
at hudson.model.Hudson.executeReactor(Hudson.java:753)
at hudson.model.Hudson.<init>(Hudson.java:667)
at hudson.model.Hudson.<init>(Hudson.java:607)
at hudson.WebAppMain$2.run(WebAppMain.java:215)
Caused by: hudson.util.IOException2: Unable to read c:\jenkins\config.xml
at hudson.XmlFile.unmarshal(XmlFile.java:152)
at hudson.model.Hudson$12.run(Hudson.java:2340)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
at hudson.model.Hudson$4.runTask(Hudson.java:735)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
My Jenkins Instance is Back Online :-)
Solution that worked for me:
I deleted the "DISABLED" files in my "C:\Jenkins->Plugins" (sort by TYPE, look for DISABLED) and Restarted my Jenkins Instance.
It loaded like a Charm. However, Configuration page Didnt load Correctly due to Some Plugins didnt get Updated good. So Identified each Job and re-loaded the Required Plugins.
Finally, used "Jenkins -> Manage Jenkins -> Reload Configuration from Disk" to get back all my Jobs back Online functioning normal.
Related
An instance of Jenkins started not saving changes made under 'Manage Jenkins > System Configuration'.
In an attempt to solve it, I have recently upgraded to Jenkins 2.346.3 (including all the plugins).
Unfortunately, this behavior still persists and the System Log only shows:
Error while serving http://<jenkins_url>/configSubmit
java.lang.ClassCastException: java.lang.Integer cannot be cast to hudson.model.Describable
at hudson.util.DescribableList.get(DescribableList.java:128)
at hudson.util.DescribableList.rebuild(DescribableList.java:170)
at jenkins.model.GlobalNodePropertiesConfiguration.configure(GlobalNodePropertiesConfiguration.java:24)
at jenkins.model.Jenkins.configureDescriptor(Jenkins.java:4017)
at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3981)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
Caused: java.lang.reflect.InvocationTargetException
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:401)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
<snippet>
Any idea on the possible cause?
UPDATE
After 2 attempts on restarting Jenkins without the config.xml, I succeeded in having Jenkins 'Manage Jenkins > System Configuration' behaving as expected.
After the first attempt, I reverted to the old configuration file as all the security related configurations were missing and I ended up raising the ticket https://issues.jenkins.io/browse/JENKINS-69548
On the 2nd attempt, I did what I described in the ticket comment https://issues.jenkins.io/browse/JENKINS-69548?focusedCommentId=430091&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-430091 (i.e. re-introducing the security-related configurations)
You probably have a corrupted config.xml from your old installation. Try deleting the config.xml(Back it up) located at $JENKINS_HOME(if you have not changed the default JENKINS_HOME in most cases it will be at USER_HOME/.jenkins(~/.jenkins)) and restarting Jenkins. If it's successful you can start reconfiguring or moving the configs from there.
I've managed to break Jenkins while trying to fix some other problems. Now whenever I run a job it immediately fails with the following error:
java.lang.NullPointerException
at java.util.TreeMap.putAll(TreeMap.java:313)
at hudson.slaves.EnvironmentVariablesNodeProperty.buildEnvVars(EnvironmentVariablesNodeProperty.java:85)
at hudson.model.Computer.buildEnvironment(Computer.java:1195)
at hudson.model.Job.getEnvironment(Job.java:385)
at hudson.model.Run.getEnvironment(Run.java:2419)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:500)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:112)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:68)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:310)
at hudson.model.ResourceController.execute(ResourceController.java:99)
at hudson.model.Executor.run(Executor.java:432)
Finished: FAILURE
I found this answer Jenkins - Problem: hudson.model.Queue$MaintainTask failed but it suggests rewriting the config.xml file. When I do that and reload config from the disk, config.xml files gets overwritten by jenkins. I have upgraded jenkins to the latest available version and also updated most of the plugins. I also tried to add back the environment variables through the UI but doing that throws the following exception:
2022-01-06 17:21:50.894+0000 [id=10] WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving https://<jenkins-url>:<port>/configSubmit
java.lang.ClassCastException: java.lang.Integer cannot be cast to hudson.model.Describable
at hudson.util.DescribableList.get(DescribableList.java:126)
at hudson.util.DescribableList.rebuild(DescribableList.java:168)
at jenkins.model.GlobalNodePropertiesConfiguration.configure(GlobalNodePropertiesConfiguration.java:25)
at jenkins.model.Jenkins.configureDescriptor(Jenkins.java:3846)
at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3810)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
Caused: java.lang.reflect.InvocationTargetException
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:400)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
at org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)
which looks similar to the first exception where it is unable to recognize the Integer field. How can I get this configuration fixed?
How it started
I had several scheduled jobs running perfectly fine. But suddenly after weeks of successful runs they started to fail without any obvious reason. But they were still running to a certain point. I also had a plugin installed called build-monitor-plugin but after installing that I had not restarted jenkins. I decided to restart jenkins to see if that resolves the problem but that landed me into a different set of issues and I never got to figure out why the jobs started to fail after running successfully for several weeks. To find that out I have to resolve the NPE issue first (which I described in the beginning). The folder that had all my jobs now shows a broken view:
However, I am still able to access the jobs using a direct url to them. This folder had a view added to it using the build-monitor-plugin and I suspected it messed up something after the restart. So I uninstalled the plugin but that did not fix the view. I have also manually removed all files related to this plugin but that also does not help. When I visit the folder I see following error in the log:
2022-01-06 05:29:47.232+0000 [id=87] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID cb5bdf8c-2a97-4360-8f80-95df2c5c719b
org.apache.commons.jelly.JellyTagException: jar:file:/app/jenkins/war/WEB-INF/lib/jenkins-core-2.319.1.jar!/hudson/model/View/index.jelly:42:43: <st:include> org.apache.commons.jelly.JellyTagException: jar:file:/app/jenkins/war/WEB-INF/lib/jenkins-core-2.319.1.jar!/lib/hudson/projectView.jelly:67:24: <d:invokeBody> `java.lang.NullPointerException
at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:281)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)`
Finally managed to fix both issues.
What happened?
Sequence of installing the plugin-in build-monitor-plugin, restarting Jenkins and then uninstalling the plug-in left some of the files in Jenkins in a corrupted state. A state from which Jenkins version 2.263.3 was unable to recover even though it kept on recreating the files but every time it was creating the same corrupted files again.
Corrupted Files
There were two files that got corrupted and both are named config.xml. First one is in the top most level directory of Jenkins. It is supposed to be as follows in case you have environment variables:
<globalNodeProperties>
<hudson.slaves.EnvironmentVariablesNodeProperty>
<envVars serialization="custom">
<unserializable-parents/>
<tree-map>
<default>
<comparator class="java.lang.String$CaseInsensitiveComparator"/>
</default>
<int>2</int> <!-- This number indicates how many key/value pairs will follow it -->
<string></string>
<string></string>
...
</tree-map>
</envVars>
</hudson.slaves.EnvironmentVariablesNodeProperty>
</globalNodeProperties>
Something similar was also suggested in Jenkins - Problem: hudson.model.Queue$MaintainTask failed but the value of <comparator> class mentioned there did not work. I figured out the correct value by installing a separate instance of Jenkins and comparing the new config.xml with the old one.
Similarly the other config.xml that was broken was for the folder which was not showing up properly as can be seen in question. For that I created another folder and compared it's config.xml with the broken one and found following two lines missing from the <folderViews>:
...
<folderViews>
...
<primaryView>All</primaryView>
<tabBar class="hudson.views.DefaultViewsTabBar"/>
</folderViews>
...
Adding these two lines resolved the issue.
Piggybacking on this answer since we just had a very similar error pop up and we had the build-monitor-plugin plugin installed, too:
java.lang.NullPointerException
at java.util.TreeMap.putAll(Unknown Source)
at hudson.slaves.EnvironmentVariablesNodeProperty.buildEnvVars(EnvironmentVariablesNodeProperty.java:85)
at hudson.model.Computer.buildEnvironment(Computer.java:1210)
at hudson.model.Job.getEnvironment(Job.java:385)
at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:334)
at hudson.model.Run.getEnvironment(Run.java:2419)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:943)
at CUSTOM_PLUGIN
I immediately thought our custom plugin was to blame but reverting to the previous stable version did not change this behavior. Then, finding #Syed Ali's answer, I looked at our config.xml files but they seemed fine.
The problem was that, in the Configuration of the administrative section, the Environment Variables section's checkbox was checked but no key-value pairs had been added. For some reason, unchecking that option fixed any related problem.
I upgraded from jenkins 2.219 to 2.272 (latest version as of this writing) and now getting the stack trace below when Jenkins starts.
Jenkins docs says that this happens due to the Configuration as Code plugin and to set the jvm args as -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000. The docs also say to increment the value until the error goes away but so far I'm at 480000 and still getting the error. I also don't see that I have the Configuration as Code plugin installed anyway.
How can this be fixed?
java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded
at jenkins.model.Jenkins.save(Jenkins.java:3379)
at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3398)
at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2637)
at jenkins.model.Jenkins$16.run(Jenkins.java:3342)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1162)
at jenkins.model.Jenkins.<init>(Jenkins.java:962)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:295)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:312)
For anyone having this issue, yes downgrading to https://get.jenkins.io/war-stable/2.263.1/ first is required. Then restart the service, then upgrade all of your plugins, then try the LTS upgrade again, at least for me it worked just fine.
More details about the issue here:
https://www.jenkins.io/doc/upgrade-guide/2.204/
SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed
ConfigurationAsCode.init java.lang.IllegalStateException: An attempt
to save the global configuration was made before it was loaded If
you encounter this, you can tell the plugin to delay configuration for
an amount of time to give Jenkins time to load the global
configuration before the configuration is applied by the plugin.
To enable this set the
io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system
property to a number of milliseconds to delay the initialisation. The
required value will be dependant on aspects of your system (cpu/disk)
and configuration, and how it can be found is mostly a trial and
error. A suggestion would be to start with 5000 (5 Seconds) and then
increment by 2000 (2 seconds) until you no longer exhibit the issue
and finally add 1000 (1 second) for some extra safety. For example, to
delay the configuration by 9 seconds you would use something like the
following command java
-Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000 -jar jenkins.war. Exactly how and where you specify this option depends on
the installation method used to install Jenkins
However for me the workaround didnt work without first downgrading, upgrading all plugins, then finally upgrading the core again.
To avoid this issue beforehand when you actually have not installed the Casc plugin:
Update to https://github.com/jenkinsci/role-strategy-plugin/releases/tag/role-strategy-3.1 and you should be OK.
When jenkins is already in the faulty state:
download the hpi and drop the hpi file into JENKINS_HOME/plugins folder.
https://updates.jenkins.io/download/plugins/role-strategy/
(information taken from: https://github.com/jenkinsci/configuration-as-code-plugin/issues/1531)
I faced the same issue. I have downloaded the jenkins.war , the earlier version (2.263.1) from https://www.jenkins.io/download/ . Stopped the jenkins and replaced the war with my install directory(C:\Program Files\Jenkins). And started the jenkins.This works like a charm.
You can download the hpi and drop the hpi file into JENKINS_HOME/plugins folder. Usually the jenkins plugin directory will be under - /var/lib/jenkins/plugins/.
The link to get Role strategy hpi plugin - https://updates.jenkins.io/latest/role-strategy.hpi. And restart Jenkins using this command: systemctl restart jenkins.
My scenario was to upgrade the Jenkins to the latest version and then restore the data in it to work like it was working before.
While upgrading from Jenkins v2.263 to v2.303, I had the same error. The only thing which worked for me was to upgrade my global plugin i.e Role-based authorization strategy first, and then Jenkins. It worked like a crisp.
We had the same issue after we upgraded to the latest version.
After some searching we decided to go with a lower version which has LTS support, 2.263.1 in this case.
That version worked instantly so I think this error is a bug.
we has similar issue, resolved by downgrading the jenkins.war file downloaded from URL: https://get.jenkins.io/war-stable/2.263.1/
I have faced similar issue, and found that we don't need to downgrade version.
Below are the steps I performed and it worked for me
upgrade global plugin i.e Role-based authorization strategy first from Jenkins Console.
Here are steps which I have performed:
#on ubuntu, in /usr/share/jenkins:
sudo service jenkins stop
sudo mv jenkins.war jenkins.war.old
sudo wget https://updates.jenkins-ci.org/latest/jenkins.war
sudo chown -R jenkins:jenkins jenkins.war
sudo service jenkins start
I have the same error on Jenkins 2.332.2 and updated plugin role-strategy v483.v17281966f5c3.
I had to download the version 3.2.0 of the plugin and put it directly in the jenkins plugins folder. (previously I remove the existing '.jpi' file)
Then restarting Jenkins /etc/init.d/jenkins restart, and the issue has gone.
Thank you for the intel!
I had a working Django setup on OpenShift with Jenkins building so this has worked before. Now, for couple of weeks I have had problems with Jenkins. Here is one symptom:
FATAL: Unable to delete script file /tmp/hudson2562951972265911713.sh
hudson.util.IOException2: remote file operation failed: /tmp/hudson2562951972265911713.sh at hudson.remoting.Channel#1826d31:stablesbldr
at hudson.FilePath.act(FilePath.java:900)
at hudson.FilePath.act(FilePath.java:877)
at hudson.FilePath.delete(FilePath.java:1262)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:101)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:802)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:584)
at hudson.model.Run.execute(Run.java:1575)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:237)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:494)
at hudson.remoting.Request.call(Request.java:129)
at hudson.remoting.Channel.call(Channel.java:672)
at hudson.FilePath.act(FilePath.java:893)
... 13 more
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
.......
So Jenkins cannot remove the remote script on the builder machine. I even re-installed the jenkins gear and re-added the Jenkins cartridge to my application.
Apparently it had something to do with the connection dropping to the slave.
The second comment on this answer: https://stackoverflow.com/a/9998520/216846 did the trick for me.
When I contsantly refreshed the node status it showed me that the response time rose as high as 400ms when Django was post processing static files. Maybe that dropped the slave to offline mode.
I deployed jenkins.war to tomcat webapps. and changed security setting to "allow only login user..." without creating any users yet.
So now I am not able to login jenkins, and can not disable the login option either.
After googling around, seems it can be solved by editing config.xml file under jenkins root directory; however, there's not such xml existing in my jenkins root dir.
How do I recover from here?
Many Thanks!
Update:
Actually jenkins is running on the Windows 7 OS.
After specifically set the environment variable as JENKINS_HOME="F:\Tools\apache-tomcat-7.0.50\webapps\jenkins"
then navigate to http://localhost:8080/jenkins/
I see below Error -
*hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.io.IOException: Unable to read F:\Tools\apache-tomcat-7.0.50\webapps\jenkins\config.xml
at hudson.WebAppMain$3.run(WebAppMain.java:234)
Caused by: org.jvnet.hudson.reactor.ReactorException: java.io.IOException: Unable to read F:\Tools\apache-tomcat-7.0.50\webapps\jenkins\config.xml
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:44)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:910)
at jenkins.model.Jenkins.<init>(Jenkins.java:809)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.model.Hudson.<init>(Hudson.java:78)
at hudson.WebAppMain$3.run(WebAppMain.java:222)
Caused by: java.io.IOException: Unable to read F:\Tools\apache-tomcat-7.0.50\webapps\jenkins\config.xml
at hudson.XmlFile.unmarshal(XmlFile.java:167)
at jenkins.model.Jenkins$17.run(Jenkins.java:2562)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.InstantiationError: hudson.security.AuthorizationStrategy
at sun.reflect.GeneratedSerializationConstructorAccessor5.newInstance(Unknown Source)*
(Answered in a question edit. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )
The OP wrote:
Issue was solved now :-)
I first changed the port of server.xml file for Tomcat
--> Connector port="new-port" protocol="HTTP/1.1"
I then removed the JENKINS_HOME environment variable from System
variables definition.
then removed deployed 'jenkins' directory under 'tomcat/webapps'
directory
and removed everything under 'tomcat/work/Catalina/localhost'
directory
then restart Tomcat server
Now I'm in Jenkins page without requiring a login - "Welcome to Jenkins! Please create new jobs to get started. "
Hopefully, this could help others who having the same issue.
In the Jenkins installation, I edited the config.xml to have <useSecurity> set to false. This enabled me to go back into Jenkins and redo my security settings.
You might need to restart the Jenkins service for this to take effect.