We're migrating from "classic" Jenkins to Kubernetes with pipelines - for now we're using scripted pipeline, but I'm not sure whether declarative ones would solve this issue for us.
On "old" Jenkins, when the test configuration failed the build was marked as unstable. I'm struggling to do it with pipelines - because by default the failed test configuration leaves the build result unchanged, which shows the successful build in the end.
The documentation mentions the flag failureOnFailedTestConfig, but that one flips the build all the way to FAILURE, which is not what we want (we leave that to compilation issues).
I would be able to work around it and switch any non-success to UNSTABLE, but that is not possible. When the build is FAILURE already, there is no way I'm aware of to lower the result to UNSTABLE. But when it's success, I have no idea, whether some test configurations failed or not.
I also checked whether the step produces any return value, perhaps I can work with that... but no, there are no stats I can use in the pipeline afterwards.
The failure* flags do not actually cause the step to fail, so I can't handle it with catchError as I like - it merely marks the result so I'm out of options.
It seems like some unstableOnFailedTestConfig flag is missing there. Or is there any other way how to get to the same effect? I've been searching the Internet for weeks now, but I couldn't find anything about this problem.
Checking skipped test count is not an option for us, we have some tests that are expected to be skipped based on environment.
Related
So I have a Jenkins job I'm testing right now that is comparing two Swagger contract files like this.
I'm using this to run my compare:
npx swagger-diff <url1> <url2>
What happens when I modify the contract on two of the endpoints I only get a warning and not an error and it wont cause Jenkins to report that the build failed. Is there anyway I can configure this to ensure that the Jenkins build will fail if there are contract differences?
The ultimate goal if this is to be able to watch an endpoint so that we know when it is failing.
So I ended up solving this by using the Jenkins plugin Log Parser to read through the log and look for specific breaking changes that we determined would be "fatal".
General Symptoms
We use Jenkins to build & test on multiple platforms. We use the Configuration Matrix plugin to help with this. Occasionally (increasingly often) Jenkins will mark the Configuration Matrix master job or subjobs as failed when the jobs seem to have succeeded (the console output reports success). We have no idea why this is happening. Any suggestions?
Some clues:
The exit code of the Jenkins job's script is not relevant. We've had test sub-jobs that simply exit 0 and they can still exhibit this bad behavior.
The failures are bunchy. They seem to come in groups.
The failures tend to effect our Windows platforms more heavily but the issue occurs on our Mac nodes as well.
The failures clump on a single node for a time but they are not exclusive to any 1 node.
We've noticed that the failures happen most often with load, particularly failed sub-jobs are started later than their successful sibling sub-jobs (often after other siblings have already completed).
We suspect that the sub-jobs are somehow considered completed by the master before they actually complete. Since they're not done the master sees them as failed. Later the sub-job really does complete (thus the console output says Success). We suspect this because we've added comments to "failed" jobs which look incomplete only to return later and see additional console logs.
It turns out that there is a bug in either the Jenkins "Set Build Name" plugin or the "Configuration Matrix" plugin. When you use them both you're subject to a few bugs. First is the symptoms I described above. Second is that the names set on builds can be wrong (race conditions mean that the wrong name can be applied).
There is a ticket open against Jenkins here. Unfortunately there isn't currently a posted work-around. We may simply stop using one of the plugins.
I'm using Jenkins to build a number of client libraries to my system, each one targeting a different language. My configuration consists of a number of distinct build steps, one for each language, building and/or testing the library.
Right now, a breaking change in the system will cause the first build step to fail, and no other build step ever gets run, which is very inconvenient. I need to run all build steps, so that I can see all resulting errors.
One alternative would be for a configuration to exist to tell jenkins not to abort latter build steps if earlier ones fail. I didn't find any such thing, but maybe there is.
Another alternative I thought of was to trigger each language as a separate build, but then I have to tie them all up for my post build steps, and I'm not sure how to do that.
And, of course, there might be something I'm missing.
Any of the above would solve my problem. So, what should I do, and how do I do it?
The Multijob Plugin is what you are looking for. I use this plugin a lot. Take a look at their screenshots - You can specify three options on how to mark a Phase(a Phase can have multiple jobs) as successful
when the Phase Job's build was successful
even if the Phase Job's build was unstable
even when the Phase Job's build failed - this is what you need
In your case, you can create a Multijob Job that has one phase for each 'client library'. And add your respective 'client library' jobs into their Phrases. So if you have 10 'client library' jobs then you will end up having 10 Phases (one for each) in this MutliJob. And don't forget to set the 'Mark phase as successful when phase jobs' statuses are' as 'Complete (always continute)'
I usually don't prefer using Downstream Jobs for this purpose, because, there is no nice UI in Jenkins to look at the whole build flow in a single page. Ofcourse there is this Build Pipeline Plugin that shows each and every build flow status. You may choose the right plugin based on your need.
To make this work in one job, I think you need to make the build steps to not fail (return non-zero exit code). Then you need a post-build step which analyzes test results and marks build failed or unstable, if needed.
You may need to write a wrapper script to do the build, such as simple .BAT or shell script, which has exit 0 as last line, after running the build/test actions.
Example of a post-build action which does this kind of test result processing is Publish JUnit test result report.
Is it possible to set the build result for a build after that build is complete?
I could not find any plugins that do this already, and I was considering writing my own, but I wanted to see if this was even possible before going down that path.
(I have looked at existing code and how the "Fail The Build" plugin works as an example, but my understanding of the Jenkins code base is not advanced enough to understand what all the possibilities are.)
Use case: we have a build pipeline, and near the end of the pipeline there is a deploy-to-qa step that deploys the artifact to a QA environment. We have automated tests before this step to try to catch any problems with the artifact, but our test coverage is not very high in some areas so bugs could still slip through the cracks. I'd like to have the ability to mark a deploy-to-qa build as FAILED after the fact, to denote that that particular pipeline was invalid and is not a candidate for production release. (Basically the same as this Build Pipeline Plugin issue)
After some more investigation in the code, I believe that this is not possible.
From hudson.model.Run:
public void setResult(Result r) {
// state can change only when we are building
assert state==State.BUILDING;
// snip
...
}
So the build result cannot change except when in "building" state.
I could try to muck with the lastSuccessful and lastStable symlinks (as is done with the delete() function in hudson.model.AbstractBuild), but then those would be reset as soon as Jenkins reloaded the build results from jobs/JOBNAME/builds/.
I have an untested suggestion: Make a parametrized build, where the parameter determines if build will fail or not (for example simple bat / shell script testing the parameter from the environment variable it sets, and doing exit 0 or exit 1). This assumes that build pipelines manually triggered step will ask the parameters, and not use default values.
If it does not support interactive build parameters, then some other way is needed to tell this extra build step wether it should fail or not. Maybe editing upstream build description or display name to indicate failure, and then allowing build pipeline to continue to this extra build step, which probably has to use system groovy script to dig out upstream build description or display name.
I have seen several debates on this topic previously, and the outcome was always that it is theoretically possible to do so, but the codebase is not designed to allow this and it would have to be a very hacky workaround.
It's also been said that this is a bad practice in general, although I don't remember what the argument against it was.
I am facing the same requirement. I haven't found an appropriate plugin, changing the build status is not just a flag but has other impacts on links (eg latest successful build etc). So instead of changing the status of the build I looked for a possibility for qualifying the build. The Promoted Builds Plugin apply flags to build to define e.g. different quality stages. Build promotions can be performed manually or based on e.g. downstream project successful builds. Any successful build can be qualified, based on the promotion additional build and post build actions can be executed, e.g tagging or archiving.
Actually I was able to do it by changing the build.xml manually to <result>FAILURE</result>.
I've then played a little bit with mklink to create some symbolic links and also renamed the lastSuccessfulBuild to lastFailedBuild and it worked. If you are allowed to access the filesystem from within a Jenkins PlugIn, then it is possible to write one.
In case you are fine to delete the current build and start the same build using a version number and setting the next BUILD_NUMBER to the deleted one, then you could use this plugin to tell it to fail instead of succeed:
Fail The Build Plugin
One team I am working on uses Jenkins + PHPUnit for CI. I am looking for a good way to indicate a particular person is addressing a particular test which may be failing the build. The Claim plugin seems great for claiming an entire failing build but I'm looking for some way to make it more granular. It would be particularly useful to make a "sticky" per-test claim so if the same test fails next build the Claimer is retained. Can Claim do this (I thought it could, but I must be missing something if so)? Can something else do this?
This is provided by the Claim plugin, as indicated by MrsTang. However, you need to activate this per job.
In the Job configuration, under the Post-Build Action Publish Test Results you will find a check box Allow claiming of failed tests. That will allow you to claimed failed tests. We found that feature to work quite fine.