I am on a redhat linux box. I recently updated Jenkins to version 1.509 only to find that after doing so it has "forgotten" two of my jobs/projects. The jobs can still be found on my Jenkins machine under /var/lib/jenkins/jobs, but they no longer show up in the Jenkins GUI. I attempted to re-create them based off the configuration file I have, but I am not confident I have totally re-created the functionality they had.
I also tried to copy the job and or rename it hoping that would get jenkins to see it, but no luck. I had tried cp -r /var/lib/jenkins/jobs/JOB1 /var/lib/jenkins/jobs/JOB2. I also restarted the service a number of times. Finally I updated all of my plugins on the off chance that was somehow related.
So my question is "How can I get Jenkins to notice these jobs?" or failing that "can I run these jobs from in the terminal?"
NOTE: I am not discouraging others from upgrading Jenkins. After I upgraded Jenkins did complain about a number of things which I didn't pay enough attention to which I believe got me into this mess in the first place.
If I were you, I would try the Jenkins CLI (from $JENKINS_URL/cli) and use the create-job command and feed the job configuration file to the cli's stdin.
If that does not help, I would inspect Jenkins log files (you are saving stdout and stderr of Jenkins somewhere, right?) for any errors or clues. If the job failed to load because of some tag that you can guess is provided by a plugin, try to remove that part from the config file.
If that does not help, I would upgrade Jenkins. I think there might be some fixes related to this in the LTS version changelog since 1.509.
And above all... if I were you, I would start making backups of the job configuration files.
I regularly back up the global config.xml, all the job config.xml files and all the plugins. Using these I can set up my Jenkins from scratch. And I do that to set up a test instance where I try any plugin or Jenkins core upgrade. If I see no problems after running a few of the trickiest builds, I know I can upgrade the production instance with much more confidence.
Related
I ran a build yesterday, hoping I would read some logs today.
I came today, and got an error 404 when trying to access the build. Strange.
Running another build, shows my build actually did run, but it is unreachable.
Is there a way to get my hands on the logs?
Notice build #10 is missing, even though it did start.
Probably a windows update is to blame for this.
The broken link is http://192.168.80.10:8080/job/Dev_git/10
More information on a run can usually be found using the context menu under Console Output. This is only accessible if you have the correct permissions set in Jenkins.
This of course does not work, if a build is missing. One reason could be that your Jenkins is configured in a way that only a certain number of historic builds are kept, see Build History Missing in Jenkins for an explanation how to deal with that.
However, your case seems to be different, because a build in the middle of the history is missing. For this, I suggest to look around in the jobs directory of your Jenkins installation where it stores all the configuration and run data.
References
https://wiki.jenkins.io/display/JENKINS/Administering+Jenkins
Where does Jenkins store configuration files for the jobs it runs?
Jenkins (on a Kubernetes node) is complaining it requires a newer version of Jenkins to run some of my plug-ins.
SEVERE: Failed Loading plugin Matrix Authorization Strategy Plugin
v2.4.2 (matrix-auth) java.io.IOException: Matrix Authorization
Strategy Plugin v2.4.2 failed to load.
- You must update Jenkins from v2.121.2 to v2.138.3 or later to run this plugin.
The same log file also complains farther down that it can't read my config file... I'm hoping this is just because of the version issue above, but I'm including it here in case it is a sign of deeper issues:
SEVERE: Failed Loading global config
java.io.IOException: Unable to read /var/jenkins_home/config.xml
I'd either like to disable the plug-ins that are causing the issue so I can see the Jenkins UI and manage the plug-ins from there, or I'd like to update Jenkins in a way that DOES NOT DELETE MY USER DATA AND JOB CONFIG DATA.
So far, I tried disabling ALL the plug-ins by adding .disabled files to the Jenkins plug-ins folder. That got rid of most of the errors, but it still complained about the plug-in above. So I removed the .disabled file for that, and now it's complaining about Jenkins not being a new enough version again (the error above).
Note: this installation of Jenkins is using a persistent storage volume, mounted with EFS. So that will probably help alleviate some of the restrictions around upgrading Jenkins, if that's what we need to do.
Finally, whatever we do with the plug-ins and Jenkins version, I need to make sure the change is going to persist if Kubernetes re-starts the node in the future. Unfortunately, I am pretty unfamiliar with Kubernetes, and I haven't discovered yet where these changes need to be made. I'm guessing the file that controls the Kubernetes deployment configuration?
This project is using Helm, in case that matters. But again, I hardly know anything about Helm, so I don't know what files you might need to see to make this question solvable. Please comment so I know what to include here to help provide the needed information.
We faced the same problem with our cluster, and we have a basic explanation about that, but not sure about it (The following fix works)
That error come with the fact that you have installed Jenkins via Helm, and their plugins through the Jenkins UI. It works if you decide to never reboot the pod, but if one day, jenkins have to make his initialization again, you will face that error.
Jenkins try to load plugins from the JENKINS_PLUGINS_DIR, which is empty, so the pod die.
To fix the current error, you should specify your plugin in the master.installPLugins parameter.
If you followed a normal install, just go on your cluster and
helm get values jenkins_release_name
So you may have something like that:
master:
enableRawHtmlMarkupFormatter: true
installPlugins:
- kubernetes:1.16.0
- workflow-job:2.32
By default, some values are "embedded" by helm to be sure that jenkins works, see here for more details: Github Helm Charts Jenkins
So, just copy it in a file with the same syntax and add your plugins with their versions. After, you have just to use the helm upgrade command with your file on your release:
helm upgrade [RELEASE] [CHART] -f your_file.yaml
Good luck !
I have Jenkins 1.6 installed as a service on a CentOS machine.
At some point in the past the service stopped/crashed/? and Jenkins was restarted from the command line, java -jar jenkins.war, as the root user.
While it was running as root some plugins were updates/installed, and jobs created created/ran. Any new files created are now owned by the root user/group and not by the jenkins user/group.
Meaning, when the service was restarted Jenkins could not read these files. Resulting in plugins not running and most jobs not being loaded.
Manually restoring the permissions(chown, chgrp) to the plugins solves the plugin related issues.
For the jobs it's easy to spot new ones and fix them. Any existing ones that were re run and created new files are more difficult to find.
Then there may be other files which the Jenkins server created, not as part of a job or plugin which need to be changed. The errors are not always obvious, and crop up slowly over time.
Am I better adding the jenkins user to the root group? could this cause issues?
Or manually changing the permissions, hope i got them all! And fix others that come up?
Any suggestions appreciated.
In the end we manually changed the file permissions back to Jenkins. There were some immediate jobs that failed in the following days. And a few which only came up a few months later.
For the most part it worked well.
I'm using Jenkins v1.546, hosted on a Windows Server 2008 R2 SP1 machine.
I've set up a fairly simple job for building a Maven Java project. It polls the SCM with no schedule and picks up remote build triggers, requiring an authentication token. It uses Subversion and performs clean checkouts with svn update. Additionally, it has a post-build step that archives some build artifacts (i.e., the resulting WAR and WSDLs).
The issue I'm experiencing is that the builds that it stores on the filesystem itself contain invalid characters in their filenames. This causes our automatic backup process to blow up, it being unable to alter or remove those directories/files with the '$'. I myself cannot move/delete those folders or files either, but if I rename it and remove the $, then things work fine. Oh, and if I try to follow one of these links with the $ in it, it doesn't resolve. None of the other jobs seem to do this - just my job, of course. Anyone know why this may be occurring and what I can do to resolve this?
I've attached multiple screenshots that show the bad filename and my Jenkins job setup. I had to white out some company information. If I can provide any additional information to help troubleshoot this, just let me know.
Also, as an update, I did some additional research, looking through the changelogs for each released version of Jenkins since my version (latest is 1.557). I saw three possible issues in the changelogs that could be related, but it's hard for me to tell. I cannot simply upgrade our Jenkins to test out this theory, since I'll need to provide a reason for upgrading beyond a hunch.
https://issues.jenkins-ci.org/browse/JENKINS-21023
https://issues.jenkins-ci.org/browse/JENKINS-20534
https://issues.jenkins-ci.org/browse/JENKINS-21958
The $ is a perfectly valid character in Windows directory name. You can manually make a folder with it, and delete it without any problems.
The com.company$moduleName syntax is used by Jenkins Maven-style job to separate modules of your build. If you don't see this structure for other people's jobs, it is because they are either not building a Maven job, or they don't have multiple modules in a single job.
What is strange though it that these are symlinks (I don't see that in my environment). It is possible that the location that is referenced by the symlink is deleted, but the link remains. In this case, you would not be able to navigate to that location through the link (this is what you are experiencing)
Is it possible that your backup software is deleting the target directories before deleting the links?
In any case, do a simple dir on the directory with the links to see what they link to. And then verify those target locations exists. If they don't, you need to figure out who/what is deleting the links' targets
Edit:
This seems to be more related to the issue that you are facing. Unfortunately, it's marked as "unresolved"
https://issues.jenkins-ci.org/browse/JENKINS-20725
The issue stems from the fact that the symlinks are referencing to targets with / instead of \
My Maven plugin (not Maven version) is 2.6. See if upgrading your Maven plugin in Jenkins will help you. Also, I am running Maven 3.2.2 from the automatic installers. Try with that, as I don't see symlinks in my modules.
I have a Jenkins Server (1.510) on Win 2008 with ~100 jobs.
After installing and then uninstalling the CloudBees (Plugin_1, Plugin_2) set of plugins + restart I have the following issues:
half of the Jobs are now missing.
many plugins are not functioning well, for example the green-balls plugin is not working and also the entry to launch the backup plugin is missing
many built in Jenkins buttons such the new "Credentials" is missing from the "Jenkins Configure" Menu.
Looking at the FS, i still see all the jobs.
I already tried :
Using the reload configuration
Reinstalling the plugins
Reinstalling Jenkins with the same version once again
Still the jobs are missing
Any idea how to solve it?
Thanks,
Doron
When job is loaded, many of the related Java classes get instantiated. If instantiation fails, usually because some plugin has been removed and that class is no longer available at all, then the job is hidden.
I suspect you have accidentally removed some other plugin too.
Note: Before actually doing anything, take full backup! Easiest is to backup entire Jenkins folder, where the jobs, configuration etc reside.
Easiest solution might be to just install Jenkins from scratch, install the plugins you do need (see below for troubleshooting if you're missing some), then copy the jobs subfolder to the new Jenkins. It might be best to do any configuration under Manage Jenkins by hand, but you can also just try copying the related XML config files.
If you are missing a plugin and can't figure out which one, you should look at jenkins.out.log and jenkins.err.log log files and search for exceptions happening after Jenkins is started. That may give you a clue on what plugin you are missing.
You can also try editing the job XML files to remove build steps you identified from the exceptions (remember to take backups first!), then restart Jenkins or select Reload configuration from disk from Manage Jenkins page.
If not solved, but you find relevant-looking exceptions or something else interesting, please update the question with details.
After I upgraded Jenkins, one of my jobs disappeared. I found out that although my job directory still existed, the config.xml file inside had somehow gone missing.
I restored this file from a backup, after updating all the plugins that needed updating, and reloaded the configuration, and the job reappeared in Jenkins.