Jenkins Gerrit polling fails: constantly triggering builds - jenkins

So we're using Gerrit Trigger (2.23.0) on our Jenkins CI build manager & using docker containers for the actual builds.
The issue that has recently popped up in some of our branches the Gerrit Repo polling is failing and causing it to "detect changes" every time, so it's constantly rebuilding despite no changes.
Checking the Gerrit Repo Polling Log for any of the affected jobs gives one of the following outputs:
Started on Feb 1, 2017 3:12:25 PM
Polling SCM changes on aosp-host
[workspace] $ repo init -u http://xxx.xxx.xxx.xxx/git/project/platform/manifest.git -b branch -m branch.xml
Get https://gerrit.googlesource.com/git-repo/clone.bundle
Get https://gerrit.googlesource.com/git-repo
fatal: Not a git repository: '/home/jenkins/workspace/.repo/manifests.git'
fatal: Not a git repository: '/home/jenkins/workspace/.repo/manifests.git'
fatal: cannot obtain manifest http://xxx.xxx.xxx.xxx/git/project/platform/manifest.git
Done. Took 1 min 19 sec
Changes found
or, if the build was already building (gerrit waits for the build to finish before doing the scm poll)
Started on Feb 2, 2017 3:24:15 AM
Polling SCM changes on aosp-host
[workspace] $ repo init -u http://xxx.xxx.xxx.xxx/git/project/platform/manifest.git -b branch -m branch.xml
fatal: cannot make /home/jenkins/workspace/.repo/repo directory: File exists
Done. Took 2 hr 4 min
Changes found
The builds, which are triggered by this failure, use the same commands and work fine:
[workspace] $ repo init -u http://xxx.xxx.xxx.xxx/git/project/platform/manifest.git -b branch -m branch.xml
Navigating to the manifest directory, we see the symptom:
jenkins#f052b3453d95:~/workspace/.repo$ ll
total 32
drwxr-xr-x 1 jenkins jenkins 180 Dec 20 11:08 ./
drwxrwxr-x 1 jenkins jenkins 778 Dec 20 11:07 ../
-rw-r--r-- 1 jenkins jenkins 20087 Dec 20 10:14 .repo_fetchtimes.json
lrwxrwxrwx 1 jenkins jenkins 20 Dec 20 10:13 manifest.xml -> manifests/branch.xml
drwxr-xr-x 1 jenkins jenkins 8 Dec 16 17:33 manifests/
drwxr-xr-x 1 jenkins jenkins 50 Dec 16 17:33 manifests.git/
drwxr-xr-x 1 jenkins jenkins 28 Dec 16 17:43 project-objects/
-rw-r--r-- 1 jenkins jenkins 7756 Dec 20 10:14 project.list
drwxr-xr-x 1 jenkins jenkins 410 Dec 16 17:46 projects/
with
lrwxrwxrwx 1 jenkins jenkins 20 Dec 20 10:13 manifest.xml -> manifests/branch.xml
highlighted in red, because the associated branch.xml is not found... so the Gerrit Log from above is accurate, it's failing to init properly. This is confirmed via a repo status in the main directory:
jenkins#f052b3453d95:~/workspace$ repo status
Traceback (most recent call last):
File "/home/jenkins/workspace/.repo/repo/main.py", line 531, in <module>
_Main(sys.argv[1:])
File "/home/jenkins/workspace/.repo/repo/main.py", line 507, in _Main
result = repo._Run(argv) or 0
File "/home/jenkins/workspace/.repo/repo/main.py", line 180, in _Run
result = cmd.Execute(copts, cargs)
File "/home/jenkins/workspace/.repo/repo/subcmds/status.py", line 130, in Execute
all_projects = self.GetProjects(args)
File "/home/jenkins/workspace/.repo/repo/command.py", line 140, in GetProjects
all_projects_list = manifest.projects
File "/home/jenkins/workspace/.repo/repo/manifest_xml.py", line 350, in projects
self._Load()
File "/home/jenkins/workspace/.repo/repo/manifest_xml.py", line 407, in _Load
self.manifestProject.worktree))
File "/home/jenkins/workspace/.repo/repo/manifest_xml.py", line 443, in _ParseManifestXml
root = xml.dom.minidom.parse(path)
File "/usr/lib/python2.7/xml/dom/minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 922, in parse
fp = open(file, 'rb')
IOError: [Errno 2] No such file or directory: '/home/jenkins/workspace/.repo/manifest.xml'
The issue is, running repo init -u <url> -m branch.xml via the command line works fine, and produces a valid repo.
Any insight one can offer for this issue?

Related

Jenkins symlinks/permalinks broken after restart

Jenkins version 2.114 is running builds but they seem to disappear from the Dashboard after a restart. I have Jenkins setup on Linux and all the log is present under
${JENKINS_HOME}/jobs/${ITEM_FULL_NAME}/builds
I have tried Reload Configuration from Disk with following set in /etc/sysconfig/jenkins
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true"
Below is build and workspace directory settings:
Workspace Root Directory :${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}
Build Record Root Directory: ${ITEM_ROOTDIR}/builds
Below you can see the build number 117 and 118, but still, the lastFailedBuild, lastStableBuild , lastSuccessfulBuild are set to -1, which doesn't exist
drwxr-xr-x 4 jenkins jenkins 4096 Apr 2 20:17 117
drwxr-xr-x 4 jenkins jenkins 4096 Apr 3 20:17 118
lrwxrwxrwx 1 jenkins jenkins 2 Apr 3 11:09 lastFailedBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 4 10:44 lastStableBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 4 03:21 lastSuccessfulBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 3 10:58 lastUnstableBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 3 10:58 lastUnsuccessfulBuild -> -1
I have also set the
Discard Old Builds Slicer - Days to keep artifacts as 30
Discard Old Builds Slicer - Max # of builds to keep as 40
I ran the jenkins-job-checker[ https://github.com/docwhat/jenkins-job-checker ] on one of the jobs and found the following :
Problem: NOTLINK: The number link builds/1 is not a symlink!
Problem: NOTLINK: The number link builds/2 is not a symlink!
Problem: NOTLINK: The number link builds/3 is not a symlink!
Proposal: Archive non-link builds/1
Proposal: Archive non-link builds/2
Proposal: Archive non-link builds/3
Really appreaciate some help
The links have been replaced with the permalinks file:
rwxr-xr-x 2 jenkins jenkins 4096 Aug 6 2018 98
drwxr-xr-x 2 jenkins jenkins 4096 Aug 7 2018 99
lrwxrwxrwx 1 jenkins jenkins 3 Jan 19 13:47 lastFailedBuild -> 649
lrwxrwxrwx 1 jenkins jenkins 3 Jan 19 13:48 lastSuccessfulBuild -> 622
lrwxrwxrwx 1 jenkins jenkins 2 Jun 14 2018 lastUnstableBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 3 Jan 19 13:49 lastUnsuccessfulBuild -> 649
-rw-r--r-- 1 jenkins jenkins 0 Jun 14 2018 legacyIds
-rw-r--r-- 1 jenkins jenkins 134 Jan 19 14:42 permalinks
jenkins#ip-1-9-2-253_jenkins:~/jobs/cron_dockers/jobs/ios/builds$ cat permalinks
lastCompletedBuild 651
lastFailedBuild 650
lastStableBuild 651
lastSuccessfulBuild 651
lastUnstableBuild -1
lastUnsuccessfulBuild 650
Make sure the values in the permalinks file are correct, these are the values that are used to generate the values in the Jenkins GUI.
I installed the plugin operations-center-context and that solved my problem
I found the solution in below link:
https://support.cloudbees.com/hc/en-us/articles/216227047-After-moving-a-job-symlinks-for-folders-became-actual-folders

Jenkins can't find artifacts

I have a jenkins running a job, after which I want it to "archive artifact", which is basically just get some files.
The Job clones a github and when ran, it generates some log files which I need.
The patter is:
logfiles/*
I added a command to list the directory to make sure the files are there, and indeed they are.
+ ls -la logfiles
total 24
drwxr-xr-x 2 root root 4096 Apr 22 23:23 .
drwxr-xr-x 8 root root 4096 Apr 22 23:23 ..
-rw-r--r-- 1 root root 0 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41.log
-rw-r--r-- 1 root root 1248 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41_shortmsg.csv
-rw-r--r-- 1 root root 2521 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41_stats.log
-rw-r--r-- 1 root root 8035 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41_trace_msg.log
Archiving artifacts
ERROR: No artifacts found that match the file pattern "logfiles/*". Configuration error?
ERROR: ‘logfiles/*’ doesn’t match anything, but ‘*’ does. Perhaps that’s what you mean?
Build step 'Archive the artifacts' changed build result to FAILURE
I don't really understand why they are not found.
Hope someone can help!
Thanks!
I finally found my problem. I had configured the job to do the git clone. But for some reason I also added the cloning on the build steps... once i removed that jenkins started getting the files properly.
I would still like to understand why this would happen...
Thanks to all

Buck build failing for "events-logs" Gerrit plugin (jar)

I am trying to build Gerrit plugin "events-log" jar ( https://gerrit.googlesource.com/plugins/events-log) via Buck build, but its failing with below error:
root#jenkins-test:~/events-log# ll
drwxr-xr-x 7 root root 4096 Jun 29 13:25 ./
drwxrwxr-x 24 root root 4096 Jun 29 13:22 ../
-rw-r--r-- 1 root root 1017 Jun 29 12:57 BUCK
-rw-r--r-- 1 root root 173 Jun 29 12:57 .buckconfig
drwxr-xr-x 3 root root 4096 Jun 29 13:25 .buckd/
drwxr-xr-x 6 root root 4096 Jun 29 13:05 buck-out/
drwxr-xr-x 8 root root 4096 Jun 29 13:27 .git/
-rw-r--r-- 1 root root 146 Jun 29 12:57 .gitignore
drwxr-xr-x 5 root root 4096 Jun 29 12:57 lib/
drwxr-xr-x 4 root root 4096 Jun 29 12:57 src/
root#jenkins-test:~/events-log# /root/buck/bin/buck build events-log:events-
log
Using buckd.
BUILD FAILED: No build file at events-log/BUCK when resolving target
//events-log:events-log.
Can somebody please tell how to fix it and build the jar.
Thanks
I think I've finally figured it out ...
What I wound up doing is checking out Gerrit and building the events-log plugin in the Gerrit tree.
git clone https://gerrit.googlesource.com/gerrit
cd gerrit
git co v2.11.2
git submodule init
git submodule update
git clone https://gerrit.googlesource.com/plugins/events-log plugins/events-log
buck build plugins/events-log
I'm pretty sure you just want /root/buck/bin/buck build //:events-log. See http://buckbuild.com/concept/build_target.html
Eventually I found src/main/resources/Documentation/build.md, with which I finally succeeded in building events-log:
git clone https://gerrit.googlesource.com/plugins/events-log
git clone https://gerrit.googlesource.com/bucklets
cd events-log
git checkout stable-2.12
ln -s ../bucklets .
ln -s bucklets/buckversion .buckversion
ln -s bucklets/watchmanconfig .watchmanconfig
buck build plugin
This builds buck-out/gen/events-log.jar.
If the version is 2.14 or later, please use bazle to build
And I also try to build v2.13 with buck, you need to modify gerrit_plugin.bucklet which is under bucklets folder
Old content:
target_suffix = ''):
from multiprocessing import cpu_count
from os import path,getcwd
New content:
target_suffix = ''):
with allow_unsafe_import():
from multiprocessing import cpu_count
from os import path,getcwd
Then execute "buck build plugin
Out put:
ruhu#CCM-P700:~/tools/events-log$ buck build plugin
Download http://repo1.maven.org/maven2/com/google/code/gson/gson/2.3.1/gson-2.3.1-sources.jar
Download http://repo1.maven.org/maven2/com/google/code/gson/gson/2.3.1/gson-2.3.1.jar
Download http://repo1.maven.org/maven2/com/google/gerrit/gerrit-plugin-api/2.13/gerrit-plugin-api-2.13.jar
Parsing buck files: finished in 1.3 sec (100%)
Creating action graph: finished in 1.6 sec (100%)
Building: finished in 03:47.3 min (100%) 19/19 jobs, 19 updated, 94.7% cache mis
Total time: 03:50.3 min

Jenkins Valgrind plugin appears to sum errors across tests

I am really happy to see that a Valgrind plugin exists for Jenkins. I use it for C/C++ code at work.
I have set it up in Jenkins (Linux Ubuntu 14.04 - the valgrind plugin version 0.22) to dump xml files.
My config is like this
I can see that I do get my memcheck files out in the xml directory
-rw------- 1 jenkins jenkins 1379 Oct 25 18:21 main.18996.memcheck
-rw------- 1 jenkins jenkins 1379 Oct 25 18:22 main.19100.memcheck
-rw------- 1 jenkins jenkins 2452 Oct 25 18:27 main.19489.memcheck
-rw------- 1 jenkins jenkins 2453 Oct 25 18:28 main.19605.memcheck
-rw------- 1 jenkins jenkins 2453 Oct 25 18:28 main.19692.memcheck
-rw------- 1 jenkins jenkins 2453 Oct 25 18:28 main.19774.memcheck
-rw------- 1 jenkins jenkins 1379 Oct 25 18:29 main.19963.memcheck
I can see that the memcheck files look fine with some "dirty underwear" such as
<error>
<unique>0xb</unique>
<tid>1</tid>
<kind>InvalidWrite</kind>
<what>Invalid write of size 4</what>
<stack>
<frame>
<ip>0x80483EB</ip>
<obj>/home/jenkins/workspace/DemoValgrind/main</obj>
<fn>main</fn>
<dir>/home/jenkins/workspace/DemoValgrind</dir>
<file>main.c</file>
<line>12</line>
</frame>
</stack>
<auxwhat>Address 0x41ae21c is not stack'd, malloc'd or (recently) free'd</auxwhat>
</error>
My problem is that the Valgrind plugin counts the sum of all errors in all of the main.*.memcheck files.
I expected a view more like this one:
https://wiki.jenkins-ci.org/download/attachments/60918012/valgrind-trend-graph.jpg?version=1&modificationDate=1336573302000
where the number of errors go up and down.
I must be configuring the Valgrind "Publish Valgrind Results" wrongly.
Is there a syntax where I can get the overview diagram (the URL just above) to match the number of errors according to the build, i.e.
not accumulate?

Jenkins svn plugin clean removal

After a simple apt-get update on ubuntu 10.11, I lost the svn polling possibility. None of my projects, that could poll svn, can anymore.
The plugin Update center shows an update is available for this plugin ( 1.34 -> 1.37 ) but update does not work, wether with web interface or cli.
Jenkins logs show a whole bunch of errors relating the subversion plugin
ATTENTION: Caught exception evaluating: it.isTagged(). Reason: java.lang.NullPointerException
java.lang.NullPointerException
at hudson.scm.SubversionTagAction.isTagged(SubversionTagAction.java:157)
[...]
25 janv. 2012 16:34:37 hudson.ExpressionFactory2$JexlExpression evaluate
ATTENTION: Caught exception evaluating: it.isTagged(). Reason: java.lang.NullPointerException
[...]
INFO: Loaded all jobs
25 janv. 2012 16:34:06 hudson.util.RobustReflectionConverter doUnmarshal
ATTENTION: Skipping a non-existent field list
com.thoughtworks.xstream.converters.reflection.NonExistentFieldException: No such field hudson.scm.SubversionTagAction.list
etc ... etc ...
I removed by hand :
drwxr-xr-x 4 jenkins nogroup 4096 2012-01-25 16:33 subversion
-rw-r--r-- 1 jenkins nogroup 2105983 2012-01-24 18:34 subversion.bak
-rw-r--r-- 1 jenkins nogroup 2105983 2012-01-24 18:34 subversion.hpi
-rw-r--r-- 1 jenkins nogroup 0 2012-01-25 16:31 subversion.hpi.pinned
-rw-r--r-- 1 jenkins nogroup 2103308 2012-01-25 16:31 subversion.jpi
but the plugin keeps being listed in the update area.
Question : what is the proper procedure to remove EVERY and ALL references to this plugin, to make a very clean reinstall ?
thanks
it seems to be common problem
https://issues.jenkins-ci.org/browse/JENKINS-12449
try these steps
Downgrade jenkins, restart
Disable subversion plugin, restart
Upgrade jenkins, restart
Enable subversion plugin, restart
Edit each build configuration, enable subversion, save.

Resources