I am trying to move my build agents to use the websocket protocol instead of JNLP. It seems very close but my agents are showing as disconnected.
If I look at the log for the agent it says:
Inbound agent connected from x.x.x.x < agent's IP
and in the agent's logs it says:
2020-05-13T16:36:33.132-04:00 INFO: http://jenkins.domain.internal/login is not ready: 503 < waiting for Jenkins to join the lb
2020-05-13T16:36:43.749-04:00 May 13, 2020 8:36:43 PM hudson.remoting.jnlp.Main$CuiListener status
2020-05-13T16:36:43.749-04:00 INFO: WebSocket connection open
2020-05-13T16:36:43.749-04:00 May 13, 2020 8:36:43 PM hudson.remoting.jnlp.Main$CuiListener status
2020-05-13T16:36:43.749-04:00 INFO: Connected
But the agent is showing as offline. I'm at a loss to troubleshoot this. If I delete the agent out of Jenkins then I see a line in the Jenkins log saying that the agent is not recognized.
It appears that with version inbound-jenkins-agent:4.0.1-1 of the JNLP inbound agent works fine. Other, later versions are having trouble. At least in our environment things started working after using this version.
See this issue: https://github.com/jenkinsci/docker-inbound-agent/issues/172
Related
I have set a Jenkins master (on a VM) and this is provisioning jnlp slaves as kubernetes pods.
In very rare occasions, the pipeline fails, with this message:
java.io.IOException: Pipe closed
at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:260)
at java.io.PipedInputStream.receive(PipedInputStream.java:226)
at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
at java.io.OutputStream.write(OutputStream.java:75)
at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.setupEnvironmentVariable(ContainerExecDecorator.java:510)
at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:474)
at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:333)
at hudson.Launcher$ProcStarter.start(Launcher.java:455)
Viewing kubernetes logs Stackdriver in Stackdriver, one can see that the pod does manage to connect to the master, e.g.
Handshaking
Agent discovery successful
Trying protocol: JNLP4-Connect
Remote Identity confirmed: <some_hash_here>
Connecting to <jenkins-master-url>:49187
started container
loading plugin ...
but after a while it fails and here are the relevant logs:
org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled slave engine reconnects.
hudson.remoting.jnlp.Main$CuiListener status
Terminated
hudson.remoting.Request$2 run
Failed to send back a reply to the request hudson.remoting.Request$2#336ec321: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel#29d0e8b2:JNLP4-connect connection to <jenkins-master-url>/<public-ip-of-jenkins-master>:49187": channel is already closed
"Processing signal 'terminated'"
.
.
.
How can I further troubleshoot this random error?
Can you take a look at the Kubernetes Pod-Events with Stackdriver? We had a similar behavior with a different CI-System (GitlabCI). Our builds where also randomly failing. It turned out that the JVM inside the Container exceeded its memory limitation and was killed by Kubernetes (OOMKilled) and the CI-System recognised this as a build error.
Have been trying for hours for setting up master-slave configuration in jenkins and getting this error when triggering the jnlp file from slave machine: "SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted".. Everything seems set correctly from my end - Java 1.8.181 in both master and slave machines, jenkins 2.147 in both machines.
Here is the entire log: https://gist.github.com/anuraagkb/13f4f226a411fe02596af66be877257d
My bad..for the Jenkins URL under Manage Jenkins-> Configure System I had entered localhost instead of the server IP which caused the issue.
I am trying to connect to a Jenkins master instance from a slave. From a connectivity standpoint, everything looks good. I am able to curl the selected "TCP port for JNLP agent" as set in "Configure Global Security" in Jenkins from where I am starting the slave node:
$ curl http://myjenkinsurl:7004/
Jenkins-Agent-Protocols: CLI-connect, CLI2-connect, JNLP-connect,
JNLP2-connect, JNLP4-connect, Ping
Jenkins-Version: 2.62
Jenkins-Session: 77c90621
Client: 10.0.0.2
Server: 172.0.0.2
However, when trying to get to start a slave node, I get this error reported on the slave node:
INFO: Locating server among [http://myjenkinsurl:7004]
May 25, 2017 12:22:12 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: http://myjenkinsurl:7004/tcpSlaveAgentListener/ is invalid: 404 Not Found
I also get the 404 Not Found error when curling http://myjenkinsurl:7004/tcpSlaveAgentListener/
This is what my relevant section of Configure Global Security looks like:
I am getting this error since upgrading to Jenkins 2.62 and it was previously working with a similar configuration on Jenkins 2.19.
Most of the similar queries I can find to this question do not have this additional port configured correctly, but the output I get from curling Jenkins on this set port, 7004 in my case, seems to indicate that this is not where the problem lies.
How do I get the tcpSlaveAgentListener URL to function?
My case, I'm running both: master and the slaves on Kubernetes.
The challenge of getting this working under a ingress apart, I was getting a similar error, and if I understood it right:
the /tcpSlaveAgentListener/ should exist under Jenkins main port (usually 8080).
This URL, you configure at Manage Jenkins -> Configuration-> Cloud / Kubernetes
Jenkins URL: http://jenkins:8080
Then, under Manage Jenkins -> Global Security-> Agents
TCP Port for inbound agents: 50000.
Here you can't use the same port as your main service for Jenkins.
I'm running also master and slaves on kubernetes
What I did to correct the error is to set up two different jenkins urls:
ingress url
Manage Jenkins > System configuration > Configure system > Jenkins location > Jenkins url: https://jenkins.local/
jenkins service url
Configure clouds > Kubernetes > Jenkins url: http://jenkins-service:8080
I'm getting this warning in Jenkins logs on start.
Feb 25, 2017 9:32:40 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
Feb 25, 2017 9:32:58 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution getCurrentHeads
WARNING: List of flow heads unset for CpsFlowExecution[null], perhaps due to broken storage
Feb 25, 2017 9:32:58 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution getCurrentHeads
WARNING: List of flow heads unset for CpsFlowExecution[null], perhaps due to broken storage
Feb 25, 2017 9:48:02 PM jenkins.branch.MultiBranchProject$BranchIndexing run
INFO: bible-server #20170225.214800 branch indexing action completed: SUCCESS in 2.4 sec
workflow-cps, which seems to be the problem, is part of the famous pipeline plugin - which I am using.
https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
It doesnt seem to be having any other unwanted side effects other than this annoying warning in the logs.
Anyone got ideas how to fix this up?
I was seeing the same things. Looking at the source code for the plugin, it appears this is related to a run of a pipeline that completes abnormally.
In my case, I had a run of a pipeline that ran all the way through but got a Java exception trying to send mail because the VM lost network connectivity. Once I deleted that failed run (not the pipeline itself), I stopped seeing those warnings in the logs.
EDIT 2
Okay, turns out this has nothing to do with TFS or MSBuilder. This is entirely a problem with SonarQube. The SonarQube Service is the one who sends back a 401 (Unauthorized) status not TFS. Since I run 5.4 I have no clue as to how I specify a SonarQube user because in Jenkins both of those fields are greyed out.
I am using Jenkins as a Windows Service and about 2 hours ago the service made a successful build. Now, out of seemingly nowhere, Jenkins keeps reporting 401 (Unauthorized) no matter which build job I try to start.
They all start a SonarQube scanner first
Run MSBuild
In order for SonarQube to read the analysis I have to run a rebuild command
Run the End SonarQube analysis step and it collects the data to put on our SonarQube portal.
What I don't understand is that the last change I made to anything, was to go delete a file from an ASP project and now none of the jobs work, even those that have nothing to do with this ASP project. All the projects are stored on our Team Foundation Server (not locally hosted).
The only thing that really changed was that we wanted the IP of the Jenkins and SonarQube services to be accessible outside of the server they are hosted on so we made two sites on the local IIS and made a DNS to point at those. Reading into the error log I first see status 302 which is a redirection, before I reach 402. When I go to "Configure Jenkins" I am told that my proxy settings failed...or something along those lines.
Any idea what might cause this behaviour?
EDIT
Here is a part of the error log:
INFO: SCM changes detected in CSharp Build Job. Triggering #1
Apr 20, 2016 11:04:08 AM com.microsoft.tfs.core.config.httpclient.DefaultHTTPClientFactory logHTTPClientConfiguration
INFO: HttpClient configured for https://omitted.visualstudio.com/, authenticating as it#omitted.dk
Apr 20, 2016 11:04:10 AM com.microsoft.tfs.core.ws.runtime.client.SOAPService executeSOAPRequestInternal
INFO: SOAP method='GetRegistrationEntries', status=302, content-length=0, server-wait=1164 ms, parse=0 ms, total=1164 ms, throughput=0 B/s, uncompressed
Apr 20, 2016 11:04:11 AM com.microsoft.tfs.core.httpclient.HttpMethodDirector processWWWAuthChallenge
INFO: Failure authenticating with BASIC #omitted.visualstudio.com:443
Apr 20, 2016 11:04:11 AM com.microsoft.tfs.core.ws.runtime.client.SOAPService executeSOAPRequestInternal
INFO: SOAP method='GetRegistrationEntries', status=401, content-length=0, server-wait=578 ms, parse=0 ms, total=578 ms, throughput=0 B/s, uncompressed
Apr 20, 2016 11:04:11 AM com.microsoft.tfs.core.TFSTeamProjectCollection getServerDataProvider
WARNING: Error getting data provider
com.microsoft.tfs.core.exceptions.TFSUnauthorizedException: Access denied connecting to TFS server https://omitted.visualstudio.com/ (authenticating as it#omitted.dk)
Try to change the Server URL from https://omitted.visualstudio.com/ to https://omitted.visualstudio.com/DefaultCollection. And use Personal access tokens or Alternate credentials for User name and password. Check the screenshot below:
Okay, I have found the solution. It's so stupid I can't believe I didn't think of it.
In my SonarQube.Analysis.xml found at
...Jenkins\.jenkins\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\MSBuild_2.0\SonarQube.Analysis.xml
I remembered that I had the username and password written and at some point I went and changed that in SonarQube from the default values to something else. That made all the builds break.