Jenkins - Another java.nio.channels.ClosedChannelException error - jenkins

I have a very simple Freestyle Jenkins job running on an agent. The job just runs the following shell code
df -h
dprintout=$(df -h | grep /media/data)
tmp="${dprintout#*T *T *T }"
usage="${tmp%\%*}"
if [ "$usage" -gt "98" ]
then
echo "Disk Usage Too High!"
exit 1
fi
Around 25% of the time, the job fails, with the following error message.
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
FATAL: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 19-49-169-6.prometheus-stack-prometheus-node-exporter.monitoring.svc.cluster.local/19.49.169.6:3929
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.FilePath.act(FilePath.java:1186)
at hudson.FilePath.act(FilePath.java:1175)
at hudson.FilePath.exists(FilePath.java:1740)
at hudson.plugins.ws_cleanup.PreBuildCleanup.preCheckout(PreBuildCleanup.java:103)
at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:75)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
at hudson.model.Run.execute(Run.java:1899)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:209)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:793)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:343)
at hudson.remoting.Channel.close(Channel.java:1494)
at hudson.remoting.Channel.close(Channel.java:1447)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:923)
at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:803)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Looking at the other SO tickets with the same error message, it looks like none of them applies to my situation.

Related

ERROR: Failed to download file:/jenkins_data/jenkins_home/caches/adoptopenjdk/LINUX/amd64/jdk8u345-b01.zip from agent; will retry from controller

Getting below error while running jenkins build,
Running as SYSTEM
Installing AdoptOpenJDK to /home/jenkins/agent/tools/hudson.model.JDK/jdk13
ERROR: Failed to download file:/jenkins_data/jenkins_home/caches/adoptopenjdk/LINUX/amd64/jdk8u345-b01.zip from agent; will retry from controller
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 172.30.16.214/172.30.16.214:32768
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1000)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.installIfNecessaryFrom(FilePath.java:1046)
at hudson.FilePath.installIfNecessaryFrom(FilePath.java:982)
at io.jenkins.plugins.adoptopenjdk.AdoptOpenJDKInstaller.performInstallation(AdoptOpenJDKInstaller.java:121)
at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:70)
at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:108)
at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:221)
at hudson.model.JDK.forNode(JDK.java:149)
at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:342)
at hudson.model.Run.getEnvironment(Run.java:2431)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:955)
at hudson.maven.AbstractMavenBuild.getEnvironment(AbstractMavenBuild.java:57)
at hudson.maven.MavenModuleSetBuild.getEnvironment(MavenModuleSetBuild.java:168)
at org.jenkinsci.plugins.ghprb.Ghprb.getEnvVars(Ghprb.java:354)
at org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatusListener.returnEnvironmentVars(GhprbUpstreamStatusListener.java:42)
at org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatusListener.onStarted(GhprbUpstreamStatusListener.java:102)
at org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatusListener.onStarted(GhprbUpstreamStatusListener.java:31)
at hudson.model.listeners.RunListener.lambda$fireStarted$2(RunListener.java:230)
at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59)
at jenkins.util.Listeners.notify(Listeners.java:67)
at hudson.model.listeners.RunListener.fireStarted(RunListener.java:228)
at hudson.model.Run.execute(Run.java:1895)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
java.io.FileNotFoundException: /jenkins_data/jenkins_home/caches/adoptopenjdk/LINUX/amd64/jdk8u345-b01.zip (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184)
at java.base/java.net.URL.openStream(URL.java:1165)
at hudson.FilePath$Unpack.invoke(FilePath.java:1082)
at hudson.FilePath$Unpack.invoke(FilePath.java:1074)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3502)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
atjava.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
atjava.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
at java.base/java.lang.Thread.run(Thread.java:829)
I tried rerunning build , not sure why it is appearing

What causes a stack overflow error when the Jenkins job DSL plugin is trying to delete some unreferenced jobs?

I removed some old jobs from the Jenkins job DSL file that was used to create the jobs. While the seed job is running and trying to process the unreferenced jobs by deleting them it fails with a stack overflow error.
Here is an excerpt from the error message:
Unreferenced items:
GeneratedJob{name='...'}
GeneratedJob{name='...'}
... about 20 more Unreferenced jobs listed here ...
java.lang.RuntimeException: java.io.IOException: Remote call on JNLP4-connect connection from ***.***.***.net/***.***.**.**:***** failed
at hudson.plugins.tfs.model.Server.execute(Server.java:237)
at hudson.plugins.tfs.model.Workspaces.getListFromServer(Workspaces.java:36)
at hudson.plugins.tfs.model.Workspaces.populateMapFromServer(Workspaces.java:45)
at hudson.plugins.tfs.model.Workspaces.exists(Workspaces.java:71)
at hudson.plugins.tfs.actions.RemoveWorkspaceAction.remove(RemoveWorkspaceAction.java:25)
at hudson.plugins.tfs.TeamFoundationServerScm.processWorkspaceBeforeDeletion(TeamFoundationServerScm.java:465)
at hudson.scm.SCM.processWorkspaceBeforeDeletion(SCM.java:245)
at hudson.model.AbstractProject.performDelete(AbstractProject.java:358)
at hudson.model.AbstractItem.delete(AbstractItem.java:775)
at hudson.model.Job.delete(Job.java:675)
at com.cloudbees.hudson.plugins.folder.AbstractFolder.delete(AbstractFolder.java:1176)
at javaposse.jobdsl.plugin.ExecuteDslScripts.updateGeneratedJobs(ExecuteDslScripts.java:460)
at javaposse.jobdsl.plugin.ExecuteDslScripts.perform(ExecuteDslScripts.java:361)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:79)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1818)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: java.io.IOException: Remote call on JNLP4-connect connection from ***.***.***.net/***.***.**.**:***** failed
at hudson.remoting.Channel.call(Channel.java:963)
at hudson.plugins.tfs.model.Server.execute(Server.java:233)
... 22 more
Caused by: java.lang.StackOverflowError
at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
... about 500 more lines like this ...
at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
ERROR: java.io.IOException: Remote call on JNLP4-connect connection from ***.***.***.net/***.***.**.**:***** failed
Finished: FAILURE
Help us localize this pagePage generated: Feb 2, 2020 2:49:10 PM CSTREST APIJenkins ver. 2.177
I attempted to fix things by deleting the jobs using the Jenkins UI. I got the same stack overflow error there on a couple of the affected jobs. I changed the source control setting from Team Foundation Server to None, saved the job, and then deleted it. That worked and the jobs are now cleared out.
I went back and ran the seed job again but it still fails with this message.
What causes this stack overflow error in Jenkins while trying to delete jobs?

Jenkins multijob show failed when all sub jobs succeed

I have a multijob which has 6 child job. During test, I found that the multijob always show failed even all sub jobs succeed. The error message is:
Finished Build : 1.0.0.1 of Job : Sub_Job_6 with status : SUCCESS
FATAL: Channel "unknown": Remote call on JNLP4-connect connection from build_machine/10.224.113.202:51903 failed. The channel is closing down or has closed down
java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from build_machine/10.224.113.202:51903 failed. The channel is closing down or has closed down
at hudson.remoting.Channel.call(Channel.java:945)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1078)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)
at com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:148)
at hudson.model.Run.execute(Run.java:1727)
at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:76)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE
I found the root cause. One of my sub job is to power off, power on the build machine( a reboot actually). After reboot, Jenkins client will re-connect to Jenkins server. The server may think it's a new conversation, the old one has stopped with some exception. So the final result will show failed.
After I removed this sub job, the multijob showed success finally.

Jenkins JNLP4-connect connection from xxx.xxx.xxx.xxx failed after the success of build procedure

I got a failure after all the build procedure were finished without any errors.
And I don't know why below error has occurred.
This error has marked the status of a job as a failure even all the tests were successfully finished.
FATAL: Remote call on JNLP4-connect connection from socautovm1.xxx.local/192.168.114.144:65033 failed
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from socautovm1.xxx.local/192.168.114.144:65033
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:955)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1078)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
at hudson.util.ProcessTree.get(ProcessTree.java:399)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Unknown Source)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from socautovm1.xxx.local/192.168.114.144:65033 failed
at hudson.remoting.Channel.call(Channel.java:961)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1078)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
It seems that the agent is down. Check in which node the job runs and check its status. You may need to restart the node or even free some space in order to get it back online if the current status is offline.
I restarted the jenkins agent service and it resolved the issue. (/service/jenkins-agent)
I was having a similar issue when trying to run a Job on Jenkins that fetches the list of containers on a Slave node.
Whenever I run the job I get the error below:
ERROR: Issue with creating launcher for agent Linux1(Docker). The agent is being disconnected
[EnvInject] - Loading node environment variables.
ERROR: SEVERE ERROR occurs
org.jenkinsci.lib.envinject.EnvInjectException: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on Linux1(Docker) failed. The channel is closing down or has closed down
Here's how I fixed it:
I simply restarted the Jenkins service on the main/master node.
For Windows servers:
Search for Services and then locate the Jenkins service and restart it.
For Linux servers:
You might need to use Systemd (sudo systemctl restart jenkins)
That's all.
I hope this helps

Jenkins slave job fails after successful job

I have 2 jenkins servers because my 2 builds have some incompatible system requirements.
I setup a new node for one of the servers and migrated the jobs from the other server and set them up to run on the node.
The node runs the job just fine and even archives the artifacts (they are linked from the job) but the job throws and exception and gets marked as a failure.
** Below is the output from the jobs **
Completed build, now archiving <-- I print this out at the end of my last build step
FATAL: Remote call on ops-1-jenkins-android-10-186.fam.io failed
java.io.IOException: Remote call on ops-1-jenkins-android-10-186.fam.io failed
at hudson.remoting.Channel.call(Channel.java:748)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:940)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:556)
at hudson.model.Run.execute(Run.java:1745)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
at hudson.util.ProcessTree.getKillers(ProcessTree.java:151)
at hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:212)
at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:557)
at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:564)
at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:488)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:952)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:943)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Thread.java:701)
Thanks Lovato!
After I updated all the plugins and jenkins things worked just fine.
We got this error: FATAL: java.io.IOException: Remote call on pm_jenprodslave_1 failed. We found the slave server was unresponsive and had to reboot it. Once it came back the build works again.

Resources