We have a situation where something causes our Jenkins master to start leaking memory at a certain time each night. No jobs are running at this time, so I'm not interested in logs of build jobs.
We would like to figure out what is happening around this time, but we cannot see much in the jenkins.out.log, since it doesn't log a lot and lacks timestamps.
Does anyone know if there is a way to turn on more detailed operational (not just errors) logging in Jenkins?
You could use the logs accessible from the UI (Manage Jenkins » System Log) and set the log level to ALL for the default log recorder in Log Levels. This log is only accessible from the UI though.
Alternatively, you could increase the log level of the regular jenkins.log by passing --debug=9 to the Jenkins command. If you're on CentOS, changing the value of JENKINS_DEBUG_LEVEL in /etc/sysconfig/jenkins has the same effect.
You could also use the Jenkins/Hudson monitoring plugin to see snapshots of memory use etc inside the JVM.
Also you could install sar (sysstat) to see how the server itself is doing over time (if Linux).
Related
Currently Jenkins master can do a "preventive monitoring" of the nodes in the "/computer/configure" page but it is only possible to monitor 2 disk spaces :
$JENKINS_HOME
Temp Space
I would like to monitor other disk spaces. For example, the /var/lib/docker (default directory where all docker images are pulled) is quickly full and causes some error on the nodes.
As we use the EC2 slaves plugin, it will be useful to be able to specify a threshold for this space size in order to put the node "offline" when the threshold is reached.. The EC2 instance will then be destroyed and another "healthy" slave will be started if needed.
Is it currently possible to achieve this in Jenkins ?
Version : Jenkins ver. 2.89.1
Thank you !
You are right that Jenkins monitors only those directories out of the box. The code handling this is at hudson.node_monitors.TemporarySpaceMonitor and hudson.node_monitors.DiskSpaceMonitor.
It would be fairly easy to implement a new NodeMonitor that would accept a list of directories to watch and would use the Jenkins plugins framework facility to take a node down if the space goes below a defined limit. I could even imagine a new plugin that would offer a series of enhanced node monitors. The plan has even been to extract all node monitors into a dedicated plugin since long, but this wasn't completed for various reasons (cf. JENKINS-26466 for details).
I wrote such a plugin at https://github.com/jenkinsci/inodes-monitor-plugin if you are looking for an example on how to use the NodeMonitor API.
When we abort one of the Jenkins Job builds running on windows slave, and then try to retrigger, that new triggered build fails with below error:
The process cannot access the file because it is being used by another process.
while deleting the workspace.
Possibly when the build job was killed, some process that it spawned was not killed along with it, and that process was holding the file opened.
Often that process will finish running on its own and when it does finish, the lock on the file will be released.
If the process does not go away on its own (for example, if it has opened a window such as the popup you see when a process crashes - and you would not be able to see such window running in the Jenkins service session) and you have access to the slave machine, you can log into it and use a tool like SystemInternals's Process Explorer (available at https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer) to find and kill the process. ProcExp is a pretty easy tool to use, is reasonably well known, and great for situations like this.
If you do not have access, you may have to ask the SysAdmins to reboot the slave - it's very likely overkill but if you don't have access to the slave the people you have to ask for help will likely not have much time to help with debugging.
This error may occur due to the performance issue, in such cases we just need to use a loop like to avoid this, because, the current thread has to close before accessing by another process.
You can use the given method to access the file to attach with mail as follows:
using(Attachment attpath = new Attachment(filepath))
{
//your code to access the attached filepath
mail.Attachment(attpath);
}
Jenkins test results screen shows only pass/failed results.
I would like to show quantitive (number/percentage/time duration etc') results, parsed from logs.
e.g. memory usage, run time of specific methods etc..
What is the best way to do so?
Thanks
good question.
i imagine you're probably looking to extend your Jenkins instance with some plugins that describe more info about the tests you've run. this plugin seems relevant, but requires some experience with JMeter (java-based performance measurment tool) to generate the output that this plugin can then read and display output from a JMeter task that you can run every time your build runs:
https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin
the 'readme' in the above plugin details page specifies how to set up a project to run JMeter( see the 'Configuring a project to run jmeter performance test:' near the bottom.
another way to do similar not so immediately tied to a specific jenkins build is to run resource monitors (like Cacti or collectd) on the machines running the tests and analyze those results post-build, but again, outside of the Jenkins context.
HTH.
Recently, in our enterprise production setup, it seems someone has tried to setup a new job / test definition by using another (copying) from identical job. However, (s)he seems to have NOT saved (and probably, am guessing here, closed the browser with the session being lost).
But the new job got saved though it was not set to stable or active; we knew about this because changes uploaded to gerrit, started failing in this newly setup partial job (because, these changes were in certain repos that met certain TDD settings).
Question: Jenkins system does not have trace of who setup the system in 'configure versions' option. Is there anyway to know the details of who setup the job / when was that done ?
No, Jenkins does not store that information by default.
If your Jenkins instance happen to be running behind an Apache or Nginx web server, there might be access logs that can help you. To find out when the job was created you could look at when its config.xml file was created/modified.
However, there are a few plugins that can add this functionality so that you won't have this problem again:
JobConfigHistory Plugin – Tracks changes in your job configurations and gives the ability to restore old versions.
Audit Trail Plugin – Keeps a log of who performed particular Jenkins operations, such as configuring jobs.
i know there are several issues with a topic close to this one. But as fas as I searched i did not found a thread/question with the same topic.
So here is the situation:
On our Jenkins Server we a many build jobs (maybe a few hundred). Some of them running on Slaves, some on the master. Now i was asked to change settings of some of them (lets say 50), so the have project based security and I had to change the slave server they are running on. Before they ran already on a Slave, but a different one.
The Problem:
In the beginning everything went fine. I changed the settings ob several jobs quick and startet to change the settings of the next job. But after some time the configuration settings began to load slower and slower. First it were a few secounds(after 10 Job), then a few more secounds (after 20 jobs), then like one minute (after 30 jobs) and now several minutes (after 40 jobs). I open every settings page in a new tap and close the tab once I finished my configurations.
My Question:
Why does it take Jenkins so long to open up the configuration page? Especially because in the beginning there was nearly no loading tim and now after I changed a few jobs i have to wait minutes for it. What could be the reason?
You need to see in the first step what is configuration of this new slave,
Issue can be memory based, so on master instance check the memory usage and check the java process, can be done with strace -p <PID> depends on your environment.
Instance can be slower due to memmory usage which is in the most cases.