I am running jenkins multi branch job, suddenly it not allow me to change the configuration changes, its keep on loading without any timeout issue.
Can you please some one help me on this ?
You could have a look at the Jenkins master machine CPU and memory. Look what is consuming them. I have seen this happening when the CPU is nearly 100 %. In this case, restarting the Jenkins process or Jenkins master machine could help.
Try to remember/ask colleagues if there are any recent changes to Jenkins master machine. We had similar issues after installing plugins.
Avoid executing jobs on Jenkins master, use slave agents.
You may need to clean up old builds if you are not doing this already.
in my case, after disabling / enabling all plugins one by one, it was the "AWS SQS Build Trigger Plugin", causing the "save / apply" buttons to move, and not be functional
Related
I have many long running jobs that take almost a day to complete. Splitting is not possible. If the network fails then all progress is lost.
How can a slave survive disconnections?
EDIT 1
I have around 300 slaves running in Windows tied to one single Jenkins instance.
Slaves are connected using the manual method java - jar slave.jar -jlnpUrl <serverUrl> <slaveName>. I cannot run them as a regular Windows service because some tests manipulate GUI elements and require a real interactive session otherwise test fail.
EDIT 2
According to Jenkins Cookbook I should be using Cygwin + OpenSSH approach instead of custom script with JLNP-connector. Could this improve stability?
Jenkins was not originally designed for builds to survive across server or slave restarts. There is a CloudBees Long-Running Build plugin that supports long-running builds but, unfortunately, it is available only for enterprise users and still beta.
I didn't find any free alternative and would suggest you to try to improve your network stability and to split your long running jobs. At least you can divide your tests on logical groups (test suites).
Jenkins now has a workflow plugin. It claims to handle "server" restart and loss-of connectivity with slave.
From the link
A key feature of a workflow execution is that it's suspendable. That
is, while the workflow is running your script, you can shut down
Jenkins or lose a connectivity to a slave. When it comes back, Jenkins
will still remember what it was doing, and your workflow script
resumes execution as if it was never interrupted. A technique known as
the "continuation-passing style" execution plays a key role in
achieving this.
(not tested at all)
Edit: Copied from #Jesse Glick's comments :
Workflow is open source and available for anyone running Jenkins 1.580.1+ or later. CloudBees Jenkins Enterprise does include a checkpoint feature, but this is not necessary simply to have a build survive slave disconnections and Jenkins restarts: that is automatic
I would like to take suggestion and inputs from wider audience regarding an issue with Jenkins master
I would like to have Jenkins setup in the following way to have the other backup so that it runs even if the master is down!
my plan for Jenkins master
The black arrows indicate that it is the primary configuration for Jenkins1 master and if it is down, we can switch apache to communicate with Jenkins2 instantly as a hot backup
It is possible and can be achieved and is helpful while performing OS upgrades or any other maintenance activities on master machine
I am planning to have it in such a way that to have a set of Jenkins master, which can be used in a manner, that the load is delegated to each master from Apache and entire Jenkins is not offline when we restart a master as other masters are also online
Please let me know a better way to achieve this if you have any
CloudBees have already addressed this issue with an Enterprise plugin for HA
The High Availability plugin, offered within CloudBees Jenkins
Enterprise, eliminates downtime due to master failures. Multiple
Jenkins masters act as backups waiting for a primary master failure.
Once a failure is detected, a backup master automatically boots up and
acts as a failover.The best part? Project work continues.
This is a paid solution but if you have a mission critical configuration then it might be worth the cost
In the job post build action, am archiving the artifacts. 90% of the time, when the jenkins job reaches this step, the slave on which it is running hangs (or) goes offline (or) the job hangs and if I kill the job it throws a "Caused by: java.lang.OutOfMemoryError: Java heap space" error.
Am running Jenkins ver 1.560.
Has anyone seen this or is aware of a fix for this? Any help is appreciated.
Thanks
It looks like you're running into https://issues.jenkins-ci.org/browse/JENKINS-22734 which started in version 1.560 and will be fixed in 1.563.
It's always a good idea to browse the Jenkins change log, especially the Community Ratings section, when you install a new version.
Whenever Hudson master will run out of space, slaves will disconnect and will have to be restarted.
You need to check Hudson master box and see how much space is allocated to the drive where hudson is running.
Another thing to note is that even if a job is running on slave, artifacts are archived always on master. So space allocation on master should be done properly.
I ran into this issue with 1.560v of Jenkins. Right now I have disabled the archiving of the maven artifacts from the "Build" section.
I'm new to Jenkins and Continous Integration, and I noticed that it supports master / slave nodes. I really don't know what that means.
Can someone please tell in what scenario do I need slave agent?
Here is a scenario:
Our main Jenkins master is running on Windows machine (yes I know... I know...). We are doing iOS mobile development. There are some things that can only be done using Xcode (which only runs on Mac OS). I have a Jenkins Slave running on that Mac, that takes care of executing those tasks that can only run on a Mac.
Why not just setup a new instance on that Mac? Cause that job is tied together with other jobs (on Master) in dependencies and the flow. Even promotions on those Xcode tasks are run on Master.
Jenkins' Master / Slave architecture is used to manage distributed builds.
There are many different scenarios you might want to use a distributed build system. It is all based on your projects load and dependencies.
Pretty much, the Master is what you're probably currently using, and is responsible for scheduling builds, dispatching jobs to slaves, and monitoring the results, but can also execute jobs itself. A slave is a java executable that sits on a remote server waiting for instructions from the master (to execute build).
To use this functionality in Jenkins, go to "Manage Jenkins" screen, and click on "Manage Nodes"
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds
For a more in depth summary of distributed builds with Jenkins, as well as some scenarios where this system would be useful, and how to implement it, please read chapter 11: Distributed Builds of Jenkins: The Definitive Guide by John Ferguson Smart
http://wakaleo.com/books/jenkins-the-definitive-guide/download-jtdg-pdf
I recently deleted 2 job from Jenkins (via GUI). When I log into slaves afterwards I still see workspace of those 2 jobs lying around. Is this behavior normal?
Notes:
Jenkins master and slaves are all running on Windows environment.
Master runs on Windows Server 2003 and slaves run on Windows Server 2008 R2.
Jenkins version is 1.509.2
Regards,
Benil
Unfortunately it is: https://groups.google.com/forum/#!topic/jenkinsci-users/SiZ3DL-UJ-8
Workspaces found on slaves are not deleted because this is a non-trivial problem (it would mean that jobs would need to record every slave the job has ever been executed on and it would also have to take into account slaves that are offline, for a real thorough solution).
I have just written a script that can be scheduled periodically to clean unused workspace. It will go through all of the Jenkins slaves and check if the directories under the workspace are already deleted in Jenkins master.
https://gist.github.com/ceilfors/1400fd590632db1f51ca
To refine what #oblio said
it would mean that jobs would need to record every slave the job has ever been executed on
Builds of jobs do record which slave they ran on, but builds can be (and often are) deleted after a while.
and it would also have to take into account slaves that are offline
Of course, but this is handled generally by the workspace cleanup feature built into Jenkins core, since it runs as a background process that deals with currently online slaves (deleting seldom-used workspaces), so any slave which is sometimes online will eventually be cleaned.
The problem is that this feature currently ignores apparent workspaces that do not correspond to a job which does not exist at the time it runs, to err on the conservative side. This commit of mine rewrote the cleanup thread to fix some other problems, but not this one.
I came across an effective script that does a good job of only cleaning when disk space gets low and brings the slave offline: https://gist.github.com/rb2k/8372402