I have set up around 10 Jenkins jobs in a windows server and those job runs for roughly around 1h each to finish.
But in it one of the job which take roughly more than 1.5h, it frequently stops with Stack overflow error.
I tried increasing Heap size to 1024m as
Jenkins throwing java.lang.StackOverflowError -- Not just on unstash suggests.
I even reduced # of executors to 5 earlier it was at 10.
I even set up JavaMelody plugin in jenkins to get a basic report and to see if the heap size overloads.
I haven't checked the JavaMelody plugin status while the jobs are running but when I checked the status afterwards I do not see any overload issues in heap size.
I check the VM also it has more than enough space in it.
But still either the job suddenly stops without any error or it shows this "Stack overflow" error.
The job configuration is not complex either,
First I try to delete files in the workspace then by using a Test.exe (tool that was developed to run API testing) and passing parameters to it then passing that standard output to a log file and finally echoing that log back in to the jenkins console. (because I wanted to save the jenkins console output to a separate log file as well)
Are there any workarounds or solutions to resolve this?
Related
My organization just updated our Jenkins instance and are now seeing problems with our pipeline jobs. All of our declarative pipeline jobs now just stop at the “[Pipeline] Start of Pipeline” and will hang there indefinitely (see screenshot below)
When looking at what the job is doing in terms of the executors, it seems to be just sitting idily on the controller node and not even in the build queue. We have 4 executors on the build in node.
Background Info: We are running our Jenkins instance on Windows Server 2012 on premises. Our recent update was to 2.362.2. I can’t remember exactly the version number we updated from, but is was over 2 years old
Does anyone have any troubleshooting steps we could try? We tried downgrading relevant pipeline plugins but that did not seem to work. Have also tried looking at logs but am unsure if I have found any relevant information. If anyone needs more context, info, I’d be happy to provide it. I just don’t know where to even begin with troubleshooting this problem
I build openembedded image with jenkins pipeline. The pipeline ends successfully (according to logs). The finished workspace has about 40 GB. The problem is that even although the pipeline finishes, the job freezes for at least several hours. I am not sure if it ever recovers, as I have always killed it (cannot afford to have jenkins blocked for several days).
I don't observe this when building something small (~ 1GB). I also don't observe it when I wipe out the entire build as the last step. And I don't have any deployment explicitly set up.
What could be wrong?
Fixed. The problem was caused by cppcheck plugin, which was left activated for all jobs on Jenkins. After disabling the plugin the big jobs end normally.
There was a question about how I figured it out. I realized that there is an active plugin that performs a post-processing on the build output, and that the build output is really, really big. So I tried disabling the plugin and it helped.
Lesson learned: Don't put any post-processing plugins under the Overview and statistics of all builds view, as they are activated for all builds, even the undesirable ones.
We are using Jenkins (on Linux) to manage our Maven builds.
Our users can create their own job and sometimes (3 or 4 times per year), they are doing a mistake and the job generates a large log file (79 GB the last time...).
I had a look to existing plugins and I didn't find something to monitor the Jenkins log size.
For example, if the log size exceeds 200MB (when the job is running), I would like to automatically stop the build.
If you developed such shell scripts or Jenkins plugins, can you share your solution?
Thanks :)
You can use the Logfilesizechecker Plugin:
This plugin monitors the size of the output file of a build and aborts the build if the log file gets too big.
Or, if this has also an impact on the runtime, the Build-timeout Plugin:
This plugin allows you to automatically abort a build if it's taking too long. Once the timeout is reached, Jenkins behaves as if an invisible hand has clicked the "abort build" button.
In the job post build action, am archiving the artifacts. 90% of the time, when the jenkins job reaches this step, the slave on which it is running hangs (or) goes offline (or) the job hangs and if I kill the job it throws a "Caused by: java.lang.OutOfMemoryError: Java heap space" error.
Am running Jenkins ver 1.560.
Has anyone seen this or is aware of a fix for this? Any help is appreciated.
Thanks
It looks like you're running into https://issues.jenkins-ci.org/browse/JENKINS-22734 which started in version 1.560 and will be fixed in 1.563.
It's always a good idea to browse the Jenkins change log, especially the Community Ratings section, when you install a new version.
Whenever Hudson master will run out of space, slaves will disconnect and will have to be restarted.
You need to check Hudson master box and see how much space is allocated to the drive where hudson is running.
Another thing to note is that even if a job is running on slave, artifacts are archived always on master. So space allocation on master should be done properly.
I ran into this issue with 1.560v of Jenkins. Right now I have disabled the archiving of the maven artifacts from the "Build" section.
We are testing out the Cloudbees Jenkins service, but all of our jobs have started to not show any console output. This seems to happen both in jobs that are successful and jobs that fail.
We have created a new job, and cannot see why it is failing since the console output appears empty.
I'm not sure if we have hit some limit in the free version of the service, or if there is some current bug in the service that is preventing the console output from being visible. Has anyone else seen this?
Restarting your Jenkins should resolve this (this is the 2nd report we've had of this behaviour recently) - please log a support job if that is not the case.
To restart, browse to https://[account].ci.cloudbees.com/restart/