I have Jenkins 1.6 installed as a service on a CentOS machine.
At some point in the past the service stopped/crashed/? and Jenkins was restarted from the command line, java -jar jenkins.war, as the root user.
While it was running as root some plugins were updates/installed, and jobs created created/ran. Any new files created are now owned by the root user/group and not by the jenkins user/group.
Meaning, when the service was restarted Jenkins could not read these files. Resulting in plugins not running and most jobs not being loaded.
Manually restoring the permissions(chown, chgrp) to the plugins solves the plugin related issues.
For the jobs it's easy to spot new ones and fix them. Any existing ones that were re run and created new files are more difficult to find.
Then there may be other files which the Jenkins server created, not as part of a job or plugin which need to be changed. The errors are not always obvious, and crop up slowly over time.
Am I better adding the jenkins user to the root group? could this cause issues?
Or manually changing the permissions, hope i got them all! And fix others that come up?
Any suggestions appreciated.
In the end we manually changed the file permissions back to Jenkins. There were some immediate jobs that failed in the following days. And a few which only came up a few months later.
For the most part it worked well.
Related
I am trying to automate Jenkins Upgrades so they do not have to be hands on. Some documentation recommends creating a batch file with instructions on the machine running Jenkins, and create a scheduled task to run the batch job. The site I found with a batch file is here, where it says:
It does delete the complete exploded war file from the deployment location, so be careful if you save any configuration files to that directory.
What configuration file would I have to worry about? No one I've talked to at my company knows of any configuration files held there, and they seem to think we have a pretty default setup, so what could I look for manually that would tell me whether or not I should be concerned?
We are running Jenkins on a Windows virtual box, I believe with Jenkins running as a service.
Alternatively, if the above method is not the easiest or best way to automate Jenkins upgrades, does anyone know a better way?
You can ignore this warning. I've never seen anything storing configuration files in that directory. It is intended to be used as a cache only.
If unsure, check your existing war directory for any files with timestamps newer than the installation time.
Here, on a busy Jenkins master, no files have been added or modified there over a period of several months (since initial war file explosion at installation time).
I followed this documentation for moving jobs between Jenkins servers. After choosing to 'Reload Configuration from Disk' I am finding that the majority of jobs were migrated however a handful of jobs are not presented within the Jenkins dashboard. Looking at $JENKINS_HOME/jobs, I see directories for jobs which did not load and confirmed the config files are valid XML. I have looked at the System Log (via the Jenkins GUI) but find nothing pertinent. Rebooting the machine and reloading configuration from disk does not resolve the problem.
Interestingly, I can see the previous build history for a specific job if I choose to create a new job with the exact same name as a job which will not load. The config.xml is overwritten when this occurs. I am hoping to preserve the job configurations without manually recreating them.
For me I found it that I had copied the jobs over with the wrong permissions - once I ran chown -R jenkins:jenkins $JENKINS_HOME/jobs it solved the issue for me
I chose to investigate by comparing the config.xml content for loaded and unloaded jobs. I eventually identified the issue:
Jobs which used cron syntax similar to H/* (as well as H/[number]) would not load
The issue stemmed from the fact that I migrated to a previous release of Jenkins. This cron functionality was introduced within 1.5.10 via story Jenkins-17311.
Removing the noted syntax from config.xml content allowed jobs to load during subsequent 'Reload Configuration from Disk' executions.
I have this strange problem in Jenkins 1.5.31 .
I had configured Maven jobs which I still see in Jenkins_Home directory. My Jenkins Windows service was not getting started even though I could still access the Jenkins on default port 8080.
When I found in event viewer found that service is trying for 8080 which it found to be in use, so I did taskkilPID using the 8080 port process and it successfully restarted the Windows Jenkins service.
However, when I opened the Jenkins URL, I saw only two of my jobs were there and rest disappeared.
Has any one faced this issue?
I checked plugins and found that Maven project plugin installed is 1.5.3.1 and 2.1 is available but it has message -Warning: This plugin is built for Jenkins 1.538 or newer. It may or may not work in your Jenkins.
I don't see those jobs in config.xml and nor in its old copy backup.
After Jenkins restarts, there is the case when some jobs are missing if the syntax in their config.xml contains errors.
Check Jenkins logs for errors (Manage Jenkins/System Log)
Fix the errors in config.xml of each job. ( path: JENKINS_HOME/jobs//config.xml)
Restart jenkins or choose option reload configurations from disk (Manage Jenkins/Reload Configuration from Disk).
When all errors are fixed the job should appear. You can validate your configurations against some online tool, f.e.: https://www.xmlvalidation.com/
Another case is when all Jenkins jobs are missing and an empty dashboard is loaded. This indicates that Jenkins home directory is wrong, so make sure to specify JENKINS_HOME correctly and restart Jenkins.
This happened to me too after a restart. But it was not because of errors in the config.xml, but because another containing folder with the same name but in different case existed on the server.
I don't know how this happened, but after copying over the jobs from the second folder and reloading the folder the job came back up again.
For e.g: the job that went missing was FOLDER-NAME/jobs/job-name
On the Jenkins server, on disk, there was both
FOLDER-NAME/jobs/job-name
folder-name/jobs/job-name
After copying the folder FOLDER-NAME/jobs/job-name to folder-name/jobs/
and reloading the folder (https://<jenkins-host>/jobs/FOLDER-NAME/reload) the job came back up.
Upgrade Jenkins to the latest version.This will fix the problem
I have a Jenkins Server (1.510) on Win 2008 with ~100 jobs.
After installing and then uninstalling the CloudBees (Plugin_1, Plugin_2) set of plugins + restart I have the following issues:
half of the Jobs are now missing.
many plugins are not functioning well, for example the green-balls plugin is not working and also the entry to launch the backup plugin is missing
many built in Jenkins buttons such the new "Credentials" is missing from the "Jenkins Configure" Menu.
Looking at the FS, i still see all the jobs.
I already tried :
Using the reload configuration
Reinstalling the plugins
Reinstalling Jenkins with the same version once again
Still the jobs are missing
Any idea how to solve it?
Thanks,
Doron
When job is loaded, many of the related Java classes get instantiated. If instantiation fails, usually because some plugin has been removed and that class is no longer available at all, then the job is hidden.
I suspect you have accidentally removed some other plugin too.
Note: Before actually doing anything, take full backup! Easiest is to backup entire Jenkins folder, where the jobs, configuration etc reside.
Easiest solution might be to just install Jenkins from scratch, install the plugins you do need (see below for troubleshooting if you're missing some), then copy the jobs subfolder to the new Jenkins. It might be best to do any configuration under Manage Jenkins by hand, but you can also just try copying the related XML config files.
If you are missing a plugin and can't figure out which one, you should look at jenkins.out.log and jenkins.err.log log files and search for exceptions happening after Jenkins is started. That may give you a clue on what plugin you are missing.
You can also try editing the job XML files to remove build steps you identified from the exceptions (remember to take backups first!), then restart Jenkins or select Reload configuration from disk from Manage Jenkins page.
If not solved, but you find relevant-looking exceptions or something else interesting, please update the question with details.
After I upgraded Jenkins, one of my jobs disappeared. I found out that although my job directory still existed, the config.xml file inside had somehow gone missing.
I restored this file from a backup, after updating all the plugins that needed updating, and reloaded the configuration, and the job reappeared in Jenkins.
I am on a redhat linux box. I recently updated Jenkins to version 1.509 only to find that after doing so it has "forgotten" two of my jobs/projects. The jobs can still be found on my Jenkins machine under /var/lib/jenkins/jobs, but they no longer show up in the Jenkins GUI. I attempted to re-create them based off the configuration file I have, but I am not confident I have totally re-created the functionality they had.
I also tried to copy the job and or rename it hoping that would get jenkins to see it, but no luck. I had tried cp -r /var/lib/jenkins/jobs/JOB1 /var/lib/jenkins/jobs/JOB2. I also restarted the service a number of times. Finally I updated all of my plugins on the off chance that was somehow related.
So my question is "How can I get Jenkins to notice these jobs?" or failing that "can I run these jobs from in the terminal?"
NOTE: I am not discouraging others from upgrading Jenkins. After I upgraded Jenkins did complain about a number of things which I didn't pay enough attention to which I believe got me into this mess in the first place.
If I were you, I would try the Jenkins CLI (from $JENKINS_URL/cli) and use the create-job command and feed the job configuration file to the cli's stdin.
If that does not help, I would inspect Jenkins log files (you are saving stdout and stderr of Jenkins somewhere, right?) for any errors or clues. If the job failed to load because of some tag that you can guess is provided by a plugin, try to remove that part from the config file.
If that does not help, I would upgrade Jenkins. I think there might be some fixes related to this in the LTS version changelog since 1.509.
And above all... if I were you, I would start making backups of the job configuration files.
I regularly back up the global config.xml, all the job config.xml files and all the plugins. Using these I can set up my Jenkins from scratch. And I do that to set up a test instance where I try any plugin or Jenkins core upgrade. If I see no problems after running a few of the trickiest builds, I know I can upgrade the production instance with much more confidence.