I tried today to configure my nodes in Jenkins with with docker 1.3.1, and Jenkins docker plugin v0.8.
The docker containers started successfully from jenkins but jenkins rejected them as the there were some echos in my bashrc.
I deleted these nodes manually from Docker server.
Now, I need to delete these nodes from jenkins and I cannot delete them. The delete operation simply hangs.
Any change to Jenkins configuration also hangs.
Any suggestions please?
Note:
tried using
1. Groovy console to delete Docker nodes
2. Tried using delete option from Manage jenkins -> nodes
3. Tried using Jenkins-cli script to delete these nodes.
Container logs
SSH connection reports a garbage before a command execution.
Check your .bashrc, .profile, and so on to make sure it is quiet.
The received junk text is as follows:
hudson.AbortException
at hudson.plugins.sshslaves.SSHLauncher.verifyNoHeaderJunk(SSHLauncher.java:854)
at hudson.plugins.sshslaves.SSHLauncher.access$100(SSHLauncher.java:134)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:698)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:691)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
[11/05/14 15:54:55] Launch failed - cleaning up connection
[11/05/14 15:54:55] [SSH] Connection closed.
Since any change in the configuration cannot be performed, I am totally stuck.
Manually removed elements from jenkins/config.xml and restarted jenkins
Related
I have my jenkins installed on windows . and have two linux node added to it.
for some reason I needed to restart the jenkins service and now my linux is not coming online.but
when I go to these nodes I can see the option to mark them offline Which I believe comes when node is actually online like below
[![enter image description here][2]][2]
How To bring these nodes back online. before jenkins restart everything was fine and I have not made any changes to slaves.
**Changes I made to jenkins.xml before restart was **
Changed the java version
<executable>C:\Program Files\Java\jdk-17.0.5\bin\java.exe</executable>
Added keystore value in pre existing argument
<arguments>-Xrs -Xmx512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Dhudson.model.DirectoryBrowserSupport.CSP="" -jar "%BASE%\jenkins.war" --httpPort=-1 --httpsPort=8000 -httpsKeyStore="jenkins.jks" --httpsKeyStorePassword=xxxxx -webroot="%BASE%\war"</arguments>
I don't think your changes had anything to do with the error. Try these options:
Go into the Linux VMs and make sure they are online
Ping the Linux VMs from the master VM to see if they are reachable
Edit the slave node configuration, check everything is as it was before, and save
Finally, remove the slave nodes and add them again
I recently had some issues regarding version upgrade for my Jenkins server. In order to update the version of the jenkins server, the first step I did was to create a backup:
sudo tar -zcvf /tmp/jenkins.tgz /var/lib/jenkins
Then, I copied the archived file, from server A and untar it on another server, server B. I can see all the files [workspace, config.xml, jobs] of server A to server B in var/lib/jenkins.
When I am logging into the jenkins box it showed:
Jenkins detected that you appear to be running more than one instance of Jenkins that share the same home directory '/var/lib/jenkins’. This greatly confuses Jenkins and you will likely experience strange behaviors, so please correct the situation.
This Jenkins:
490566619 contextPath="" at 8779#jm1597185631ybr.cloud.phx3.gdg
Other Jenkins:
1998724099 contextPath="" at 20292#jm1584048540yxl.cloud.phx3.gdg
So, I stopped the jenkins service using:
sudo service jenkins stop
Then I restarted the service using
sudo service jenkins restart
All the jobs started to appear suddenly. I have following questions:
Why did the jobs started to show up and not throw the error of running multiple instances?
If version is the only issue, why cannot the newly provisioned
server have the updated version? Is it when I copy the files from server A, the server B files gets overwritten and hence, shows the same error of the version upgrade?
I've had this problem where I send a job to one of my nodes and sometimes the job would stuck on
[EnvInject] - Loading node environment variables.
The issue started when I moved my Jenkins machine from Amazon EC2 to Azure VM. Everytime it occured, I have stopped and relaunched the agent on the node, just so in a few minutes/hours it would freeze again. Usually I would stop the job after a few minutes, but if I let it run, in ~15 minutes I would receive the following:
FATAL: java.io.IOException: Unexpected termination of the channel
I have tried the following solutions, but with no help:
Rename the job
Create a new job (which runs just a basic command)
Uncheck the 'Use TCP_NODELAY flag on the SSH connection'
Adding '-Dhudson.slaves.ChannelPinger.pingInterval=2' to the JVM options
My architecture at the time was:
Jenkins as docker container on Azure VM (Ubuntu 18.04)
Jenkins ver 2.202
SSH Build Agents plugin version 1.31.1
Node on MacOS Mojave
The solution that sort of solved the problem, (now it occurs once in a few days, or even better) was adding ClientAliveInterval 120 to /etc/ssh/sshd_config and relaunch the com.openssh.sshd
sudo launchctl stop com.openssh.sshd
sudo launchctl start com.openssh.sshd
Jenkins master is running on Amazon instance and slave machine set up on dedicated Soyoustart machine. Worked fine until it was needed to redo the slave setup: reinstalled the OS, installed Java, added masters key to slave authorized_keys and removed/added again the slave in masters known_hosts. Set up new credentials for the slave and configured the node in Jenkins master but it is unable to connect to the slave.
The setup is the the same that is and has been working with other slaves without hiccups. The only thing different is that this time the new slave is the same machine with the IP as the old one was.
It is possible to ssh into the slave from master from CLI(replaced filename and slave IP with placeholder for this post):
$ ssh -i <key-file> jenkins#<slave-ip>
Credentials have been set up :
Node is configured:
Output when connecting to the slave:
[05/17/15 07:30:31] [SSH] Opening SSH connection to <slave-ip>.
Key exchange was not finished, connection is closed.
ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
java.lang.IllegalStateException: Connection is not established!
at com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:1030)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.getRemainingAuthMethods(TrileadSSHPublicKeyAuthenticator.java:88)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.canAuthenticate(TrileadSSHPublicKeyAuthenticator.java:80)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:207)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:169)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1173)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:701)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:696)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[05/17/15 07:30:31] Launch failed - cleaning up connection
[05/17/15 07:30:31] [SSH] Connection closed.
Version numbers:
Jenkins 1.613
SSH Credentials Plugin 1.11
SSH Slaves plugin 1.9
For those who prefer to dig into code:
SSH Credentials Plugin
SSH Slave Plugin
Trilead SSH
Am I missing something obvious here? What could be causing this? Any known workaround? Or does it look like a bug that needs to be reported?
Please let me know if more information is needed.
I'm running Jenkins master using official Docker image which uses OpenJDK8 and should not need to install JCE.
Apparently this is an unresolved issue in Jenkins/SSH security.
My current workaround is by commenting out MACs and KexAlgorithm line in /etc/ssh/sshd_config of Jenkins Slave and restarting the sshd (service ssh restart on Ubuntu)
UPDATE: the issue has been resolved as of 2017-04-29
I suspect that you need to install the Java Cryptography Extension for your JVM.
Without that the RSA key size is limited and authentication is not being established.
See https://issues.jenkins-ci.org/browse/JENKINS-26495 for more details.
We have a master Jenkins running on a Linux system. The same master is attached as node using "Launch slave via execution of a command on the master". It has the same FS root as the JENKINS_HOME. The command is ssh "machine_name" "shell_script"
The shell script gets the latest slave.jar and runs it.
The master has 0 executors. The node has been given 7. I'm seeing weird behavior in the builds, like workspaces being deleted once a day, etc. I'm not sure if this is related to the way the Jenkins Master-slave is configured.
Any ideas if this is a supported configuration?