I have two projects and with dependencies so that project A is started, it updates files from git and then runs a multi-configuration project B, which:
has three axes: "foo", "bar" and "baz" with 11 x 4 x 2 items
I'm going to call the values like fooN for item N from axis foo, etc.
has a configuration filter, ruling out the last axis by running only when
baz=="baz1" (maybe in a later phase we'll want to run also tests with baz2
for baz)
runs a shell script that only cds and calls python interpreter with a script
cd /path/to/scripts
python test_${bar}.py
So when the project is run, I expect 44 configurations to be tested. But only 43 are.
It's always the same configuration (which happens to be the last one triggered, as Jenkins seems to remember the order(?)) that does never run at all:
in the final matrix looks as a gray dot with "Disabled" tooltip
in Console output, after saying "Triggering bazN,barN,fooN" for all 44 combinations,
then "bazN,barN,fooN completed with result SUCCESS" for all except the last one, but
the last one seems to be always cancelled/aborted:
baz1,bar7,foo3 appears to be cancelled
baz1,bar7,foo3 completed with result ABORTED
Console output for the single combination is not available---it looks like it never
has been built
Jenkins log does not show anything interesting about "baz1,bar7,foo3"
What does this mean? Any other pointers how to troubleshoot this?
Edit: I tried adding a "HTTP ping" script to the repo and called it from above script,
just before the python test_${bar}.py part. Which proved that for the affected
configuration, Jenkins does not even run those lines.
Without knowing how you got here to begin with (probably a bug):
Append configure to the URL of the disabled configuration, and in the resulting form, uncheck Disabled and Save.
Not really an answer, but as a workaround to the problem, cloning whole project to a new one helped: with the new project, all configurations ran normally.
This is a solved Jenkins issue:
https://issues.jenkins-ci.org/browse/JENKINS-19179
By the Matrix Project plug-in, version 1.4:
https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin
Related
I would not have believed this if I had not seen it with my own eyes. In any sort of build step (we have tried multiple types) the term http(s)://api.DOMAIN.com is replaced with https://licensing.domain.com.
We have verified the behavior on two different computers (one Mac, one Windows 10).
Both licensing.domain.com and api.domain.com are valid domain names. licensing.domain.com is used in other jobs.
It occurs at the time of saving with any job that we create or edit. If we create a job and add a build step that should print https://api.domain.com it prints https://licensing.domain.com.
It occurs with any type of build step.
The text is modified in the configuration of the job build step.
The job does not have to be executed. It occurs when it is saved at the time of creation or when it is edited.
https://api.domain.com => https://licensing.domain.com
But when we join the protocol (HTTP or HTTPS) to the domain name, we do not experience the issue.
'https://' + 'api.domain.com' => https://api.domain.com
Changing the domain extension does not change the string.
http://api.domain.net => https://api.domain.net
I was intimately involved in setting up the system and I have not created anything that to my knowledge would cause this to occur.
I have a weird situation with Jenkins... We've just started using Gradle for a project at my job and when I run the tests locally with JUnit everything is fine. But when these tests are run by jenkins for the builds of branch "A", only one test fails because of an assert(always the same test).
org.junit.ComparisonFailure: expected: "E[ZZ0530]Z" but was:"E[SY5654]Z"
It looks like the mock is not injected or the mock is ignoring the "when" mocking statement.
Here is the test :
#Test
public void testEvent() {
Date eventDateTime = TimeUtils.parseDate("2013-05-30 00:00:00");
event.setEventDatetime(eventDateTime);
//Mocking the prefix return
Mockito.when(eventCodeHelperMock.getEventCodePrefixFromEvent(event)).thenReturn("EZZ");
//Tested methode
eventWrapper.setSuffix("Z");
// Event code = prefix + date + suffix
assertEquals("EZZ0530Z", event.getEventCode());
}
What is a lot stranger is that when I create a branch "B" from branch "A" all the tests succeeds when the build is created on jenkins.
I've made some research and tried to force an other build, wipe out the current workspace and recreating the job but it didn't work.
Thanks for your help!
I have had similar problems in the past and it has been due to the order in which the junits tests are run. For example, one test modifies the state of an object but you dont see the effects of this till the tests run in a different order, and tests unexpectedly fail. There is not sufficient code in the question you have posted to tell whether this is definitely the case, but I would recommend checking the order in which the tests are being run, and also look at the objects that you are using to determine if there is a problem with the state of those objects being 'dirtied'.
I am using the vNext build system of TFS 2015.
I currently have the my builds versioning in the traditional format. Major.Minor-rev.RevisionNumber. So, if I have a build for Major 1, Minor 12, the build version would look like 1.12-rev.1 when I start. I would like to know if it is possible to have the build version start at a number other than one, say 55. Such that the build version would look like 1.12-rev.55, and then increment by one as usual after that.
Actually, it is possible to effect this in a vNext build without hacking the database.
There are 2 steps.
First, you will need to implement a powershell build step (as the first step of the build) with the following inline script:
#Set the BuildNumberOffset. (Change this to the difference between the TFS build number,
#and the number that your build needs.)
$BuildNumberOffset = 543
#Don't change
$BuildNumberParts = $($env:BUILD_BUILDNUMBER) -split '\.'
$TFSRevision = [int]$BuildNumberParts[$BuildNumberParts.Length-1]
$BuildNumberParts[$BuildNumberParts.Length-1] = ($TFSRevision + $BuildNumberOffset).ToString()
$BuildNumber = [string]::Join(".", $BuildNumberParts)
Write-Host "##vso[build.updatebuildnumber]$BuildNumber"
Second, on the Label format field of the repository tab set the label format to "$(Build.BuildNumber)" instead of "$(Build.DefinitionName)$(rev:.r)". This is important so that your label will be the same as your updated build number.
There is a way to do this. It isn't pretty, but it works.
Assuming you have a build number format of something like $(Major).$(Minor)-rev$(rev:.r)
To do this queue up a build which will get the number 1.12-rev.1. Then go to the TFS database and into a table called tbl_Build. Find that last build you did and change the value in the BuildNumberRevision column to 54.
The next build that fires off will now be 1.12-rev.55
Unfortunately, it's impossible.
Every build definition has a build number format field where you can use some macros to dictate what the resulting build number should look like. In this format we are using $(Rev:.rr) Its start by one.
What is $(Rev:.rr)?
To ensure that every completed build has a unique name. When a build
is completed, if nothing else in the build number has changed, the Rev
integer value is incremented by one.
Source:MSDN
Moreover, if you want to generating a custom build number without increment.
Here is a blog with detailed procedures:Generate custom build numbers in TFS Build vNext
Couldn't we just manually set this in the format for one build i.e:
$(Major).$(Minor)-rev$(rev:.54)
and then afterwards revert back to:
$(Major).$(Minor)-rev$(rev:.r)
Not tried it, but if it works it'll save hacking around in the database.
You can do this easily, but only if you are using a Git repository in conjunction with a tool called GitVersion
It is a wonderful tool that I always use with my git repos. For your use-case: When you have version 1.2.3 and you want to jump to version 1.2.55, you just add a git tag 'v1.2.55' and it will start the versioning from there. GitVersion is a lot more complicated and does a lot more, but that is one of the features. You don't have to mess with special PowerShell scripts or anything, it instead reads your git repo history and git tags overrides the calculated versioning. There is already a TFS/VSTS/Azure Devops extension called GitVersion that works great from the same developer.
The Answer of #Steve Sims works still with TFS 2017 vNext. Thanks a lot!
I had only to do the first step "to add the script as an inline powershell script in my build." Thanks to #PainElemental
With this "Build number format" in the Options-Tab it works:
$(BuildDefinitionName)_1.2.0$(rev:.r)
I didn't label my sources with the build, so I don't checked:
Second, on the Label format field of the repository tab set the label
format to "$(Build.BuildNumber)" instead of
"$(Build.DefinitionName)$(rev:.r)". This is important so that your
label will be the same as your updated build number.
I think, you can edit the Label Format on the "Advanced" GetSources Options. (usually hidden).
This was also very painful for us, migrating from a existing CI system with it's own build numbering, we needed build numbers to increment from a specific value. Hacking databases wasn't allowed in the organisation, offsets seemed a cludge.
In the end, we used an AutoIt script to start and stop builds and delete the build result using the WebUI and left it running. Not nice, but it did the job.
Needs tweaking for screen resolutions and such, timings also perhaps. Use AutoIt Window Info to find the button locations, make sure browser is fullscreen (not windowed) Run it for a few cycles to ensure it robust before setting the loop larger.
#include <AutoItConstants.au3>
;Increment TFS build count (Chrome browser buttons locations). Start from build result page.
For $i = 1 To 15 Step 1
ConsoleWrite ( "Loop " & $i & " of 5" & #CRLF )
Sleep(200)
MouseClick($MOUSE_CLICK_PRIMARY, 1800, 200, 2)
Sleep(500)
MouseClick($MOUSE_CLICK_PRIMARY, 1150, 881, 2)
Sleep(2000)
MouseClick($MOUSE_CLICK_PRIMARY, 1800, 200, 2)
Sleep(500)
MouseClick($MOUSE_CLICK_PRIMARY, 1055, 165, 2)
Sleep(10000)
Next
TFS/Devops is a really immature CI system, and it's not a patch on what we were running. Unfortunately corporate policy said we move to TFS, as nobody ever got fired for buying Microsoft products (but plenty of people should have got fired for buying them/forcing them where they don't belong).
I was wondering if any of you guys had any experience generating code coverage reports in TFS Build Server 2010 while running NUnit tests.
I know it can be easily done with the packaged alternative (MSTest + enabling coverage on the testrunconfig file), but things are a little more involved when using NUnit. I've found some info here and there pointing to NCover, but it seems outdated. I wonder if there are other alternatives and whether someone has actually implemented this or not.
Here's more info about our environment/needs:
- TFS Build Server 2010
- Tests are in plain class libraries (not Test libraries - i.e., no testrunconfig files associated), and are implemented in NUnit. We have no MSTests.
- We are interested in running coverage reports as part of each build and if possible setting coverage threshold requirements for pass/fail criteria.
We 've done it with NUnit-NCover and are pretty happy with our results. NUnit execution is followed by NUnitTfs execution in order to get our testing results published in the Build Log. Then NCover kicks in, generating our code coverage results.
One major thing that poses as a disadvantage is fact that setting up the arguments for properly invoking NCover wasn't trivial. But since I installed it, I never had to maintain it.
Two things could pose as disadvantages:
NUnitTfs doesn't work well with NCover (at least I couldn't find a way to execute both in the same step, so (since NCover invokes NUnit) I have to run Unit tests twice: (1) to get the test results and (2) to get coverage results over NCover. Naturally, that makes my builds last longer.
Setting up the arguments for properly invoking NCover wasn't trivial. But since I installed it, I never had to maintain it .
In any case, the resulting reporting (especially the Trend aspect) is very useful in monitoring how our code evolves within time. Especially if you 're working on a Platform (as opposed to short-timed Projects), Trend reports are of great value.
EDIT
I 'll try to present in a quick & dirty manner how I 've implemented this, I hope it can be useful. We currently have NCover 3.4.12 on our build server.
Our simple naming convention regarding our NUnit assemblies is that if we have a production assembly "123.dll", then another assembly named "123_nunit.dll" exists that implements its tests. So, each build has several *_nunit.dll assemblies that are of interest.
The part in the build process template under "If not disable tests" is the one that has been reworked in order to achieve our goals, in particular the section that was named "Run MSTest for Test Assemblies". The whole implementation is here, after some cleanups to make the flow easier to be understood (pic was too large to be directly inserted here).
At first, some additional Arguments are implemented in the Build Process Template & are then available to be set in each build definition:
We then form the NUnit args in "Formulate nunitCommandLine":
String.Format("{0} /xml={1}\\{2}.xml", nunitDLL, TestResultsDirectory, Path.GetFileNameWithoutExtension(nunitDLL))
This is then used in the "Invoke NUnit"
In case this succeeds & we have set coverage for this build we move to "Generate NCover NCCOV" (the coverage file for this particular assembly). For this we invoke NCover.Console.exe with the following as Args:
String.Format("""{0}"" ""{1}"" //w ""{2}"" //x ""{3}\{4}"" //literal //ias {5} //onlywithsource //p ""{6}""",
NUnitPath,
Path.GetFileName(nunitDLL),
Path.GetDirectoryName(nunitDLL),
Path.GetDirectoryName(Path.GetDirectoryName(nunitDLL)),
Path.GetFileName(nunitDLL).Replace("_nunit.dll", ".nccov"),
Path.GetFileNameWithoutExtension(nunitDLL).Replace("_nunit", ""),
BuildDetail.BuildNumber)
All these run in the foreach loop "For all nunit dlls". When we exit the loop, we enter "Final NCover Activities" & at first the part "Merge NCCovs", where NCover.Console.exe is executed again - this time with different args:
String.Format("""{0}\*.nccov"" //s ""{0}\{1}.nccov"" //at ""{2}\{3}\{3}.trend"" //p {1} ",
Path.GetDirectoryName(Path.GetDirectoryName(testAssemblies(0))),
BuildDetail.BuildNumber,
NCoverDropLocation,
BuildDetail.BuildDefinition.TeamProject
)
When this has run, we have reached the point where all NCCOV files of this build are merged into one NCCOV-file named after the build + the Trend file (that monitors the build throughout its life) has been updated with the elements of this current build.
We now have to only generate the final HTML report, this is done in "Generate final NCover rep" where we invoke NCover.reporting with the following args:
String.Format(" ""{0}\{1}.nccov"" //or FullCoverageReport //op ""{2}\{1}_NCoverReport.html"" //p ""{1}"" //at ""{3}\{4}\{4}_{5}.trend"" ",
Path.GetDirectoryName(Path.GetDirectoryName(testAssemblies(0))),
BuildDetail.BuildNumber,
PathForNCoverResults,
NCoverDropLocation,
BuildDetail.BuildDefinition.TeamProject,
BuildType
)
In Jenkins there is a possibility to create free project which can contain a script execution. The build fails (becomes red) when the return level of the script is not 0.
Is there a possibility to make it "yellow"?
(Yellow usually indicates successful build with failed tests)
The system runs on Linux.
Give the Log Parser Plugin a try. That should do the trick for you.
One slightly hacky way do do it, is to alter the job to publish test results and supply fake results.
I've got a job that is publishing the test results from a file called "results.xml". The last step in my build script checks the return value of the build, copies eihter "results-good.xml" or "results-unstable.xml" to "results.xml" and then returns a zero.
Thus if the script fails on one of the early steps, the build is red. But if the build succeeds its green or yellow based on the return code it would have retunred without this hack.