Get the number of times a test has failed in Jenkins - jenkins

Question

We have some tests that fail periodically for no reason, mainly, JUnit times out. I want to know if I can get the number of times each test has failed. With that, I can see if it is certain tests that keep have issues, or it is not tied to tricky tests and more an issue with the stability of Jenkins on that server.

Answer 1

I encountered the same problems and we made a python that can grab the failings tests in the last N builds :
# -*- coding: utf-8 -*-
#! /usr/bin/python
import urllib
import re
import sys
project = "HERE_THE_PROJECT_NAME"
jenkin_host = "http://path.to.your.jenkins/jenkins/job/%s" % project
last_build = int(re.search("%s #(\d+)" % project, urllib.urlopen(jenkin_host + "/rssAll").read()).group(1))
start_build = last_build
nb_build = 200
REG_EXP = """All Failed Tests(.*)All Tests"""
FAILURE_REG_EXP = """javascript:hideStackTrace\(([^<]*)\)"""
all_failures = {}
last_seen = {}
print "Loading %s builds starting from build number %s" % (nb_build, start_build)
build_ok = 0
for build_id in range(start_build - nb_build, start_build + 1):
test_page = jenkin_host + "/%s/testReport/" % build_id
sys.stdout.write(".")
sys.stdout.flush()
failures = set()
for line in urllib.urlopen(test_page).readlines():
line_piece = re.search(REG_EXP, line)
if line_piece:
piece = line_piece.group(1)
match = re.search(FAILURE_REG_EXP, piece)
while (match):
failures.add(match.group(1))
match = re.search(FAILURE_REG_EXP, piece[match.start():match.end()])
if not failures:
build_ok += 1
for failure in failures:
all_failures[failure] = all_failures.get(failure, 0) + 1
last_seen[failure] = build_id
print
print "Done (found %s build OK)" % build_ok
nbs = [ x for x in list(set(all_failures.values())) if x > 1]
nbs.sort(reverse=True)
for i in nbs:
for test, nb in all_failures.iteritems():
if nb == i :
print "%d : %s (last seen : %s)" % (nb, test, last_seen[test])
And I obtain :
Loading 200 builds starting from build number 11032
.........................................................................................................................................................................................................
Done (found 148 build OK)
8 : 'one failing test' (last seen : 10906)
7 : 'another-failing-test' (last seen : 11019)