Cucumber test passed, but jenkins failed - jenkins

I have a lots of tests on several server. I use Jenkins to manage all of them.
On one server (Slave Windows), when I launch a test in cmd, I got something like :
c:/tests/cucumber --tag #dev -p ie
...
1 scenarios (1 passed)
12 steps (12 passed)
0m31.761s
echo %errorlevel%
0
No error in the tests, and cucumber seems good.
When jenkins launch exactly the same tests, I get :
c:/jenkins_folder/cucumber --tag #dev -p ie
...
1 scenarios (1 passed)
12 steps (12 passed)
0m28.453s
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
The test passed, but is marked failed by jenkins.
The command "echo %errorlevel%" is aborted : the job fail before this point.
The same job played on another slave work.
Same problem with all profile and all tags.
Same problem when I replace profile by real value
I don't use the --strict flag
Jenkins, plugins : all up-to-date
Code of the windows batch :
cd /test8folder
cucumber --tag #dev -p ie
echo %errorlevel%
What did I missed ?

There doesn't seem to be anything wrong with the configuration.
The fact that you get Build step 'Execute Windows batch command' marked build as failure after your cucumber command without actually seeing the echo %errorlevel% executed only reaffirms that there was an error in cucumber (more on that later).
However, in Execute Windows Batch Command, even a command in error would not exit the batch script (unlike Jenkins's default Execute Shell implementation). You should be seeing at least some exit code, 0, 1, or anything.
The only time this would happen is if something within your buildstep executed exit /b [exit_code_num]. I don't know "cucumber", but if that is in-fact a cucubmer.bat and inside there is an exit /b statement, this is what's causing it drop out of the buildstep without continuing.
Solution
You can use call cucumber [whateverparams] so that even when it quits with exit /b, the control will return to the calling process, the Execute Windows Batch command script.
Try that first. And you will probably see that your echo %errorlevel% will probably return a non-zero value when executed under Jenkins, but at least you will see it now.
Now, as for why it succeeds on command prompt, but fails within Jenkins, there could be a lot of reasons, the most common one being environment variables and paths. We can tackle that later once we actually see the exit code of cucumber. You also said it worked on another node: even more reason to believe this is an environment issue, maybe a non-existent folder...
Edit:
The reason that even "successful" test execution exits the calling script is because exit /b 0 would still quit the calling script, even though cucumber exited with "success" 0

Related

Jenkins build won't stop at all after SUCCESS

I have a Jenkins project with a job that gets triggered whenever something new is pushed on the jenkinstest branch of the associated GitHub repo.
What does the job, specifically? It builds a Visual Studio Solution on Debug and Release (both x86 and x64). So, it runs MSBuild.exe four times. After that, it runs a Google test command using an utilities_test.exe executable file.
These is the command that is executed at the first step:
echo "Building solution file on Debug x64"
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\MSBuild.exe" "D:\Jenkins\workspace\CFT_TCA_IMS\SAB_Test\Repository\Utilities\solution_windows\utilities.sln" /p:Configuration="Debug" /p:Platform="x64"
IF %ERRORLEVEL% EQU 0 (echo "Build status of the solution file with <Debug x64> configuration is PASSED") Else (echo "Build status of the solution file with <Debug x64> configuration is FAILED")
(there are three more batch scripts for every platform). The test that follows looks like this:
echo off
echo "Running test on Debug x64"
"D:\Jenkins\workspace\CFT_TCA_IMS\SAB_Test\Repository\Utilities\solution_windows\x64\Debug\utilities_test.exe" --gtest_filter=CategoryName.MyTest
exit 0
Everything seems to work fine. The build process is triggered on push, and the test seems to pass, but there's one problem. I also use Jenkins' Log Parser and at the Post-build Actions, I've added a Console Output (build log) parsing action, that uses a project rule looking like that:
info /0 Error/
info /0 Warning/
info /(?i)errorreport/
error /(?i)error/
error /(?i)failed/
warning /(?i)warning/
info /(?i)building/
info /(?i)done/
Because of this, my build seems not to stop. It usually takes around 2-3 minutes for my solution to build, but I've waited like 20 minutes and nothing happened.
The solution is getting built, and the test passes, but the build process does not stop. The last output of my console log is:
13:35:10 Note: Google Test filter = CategoryName.MyTest
13:35:10 [==========] Running 1 test from 1 test case.
13:35:10 [----------] Global test environment set-up.
13:35:10 [----------] 1 test from CategoryName
13:35:10 [ RUN ] CategoryName.MyTest
13:35:10 [ OK ] CategoryName.MyTest (0 ms)
13:35:10 [----------] 1 test from CategoryName (0 ms total)
13:35:10
13:35:10 [----------] Global test environment tear-down
13:35:10 [==========] 1 test from 1 test case ran. (0 ms total)
13:35:10 [ PASSED ] 1 test.
There's nothing about my log parser outputted here.
When I remove the Log Parser post-build action, everything seems to work fine. Also tried to remove everything from the parser's config file, but nothing gets fixed.
Any idea about what it could be? Any help would be appreciated.
Thank you.
Does your build run on a slave ? If so, Jenkins copies log files and other artifacts back to the master after completing the build steps but before marking the build as complete. You may have a network or I/O bottleneck here.
If you can't figure out the root cause and just want to have the build terminate without intervention, you can use the build timeout plugin.

In Jenkins is possible to mark an aborted job as success?

In a job where i call terraform apply, after the deploy the shell is stuck and the job doesn't end so i have to abort it. Is it possible to mark the aborted job as success or there is another way to end this job without force?
On the job summary/artifacts/changes/triggers/results page you can "add a description". This is a good place to summarize results and explain what happened after job result is reviewed. The first part of description also appears in Build History.
Answer:
In shell scripts (or jenkins job script) you can use bash timeout command to protect any command that might get stuck. 'timeout [<option>] <duration> <command>'. e.g.
$ timeout 2s sleep 4
$ echo $?
124
You can check this exit value in script and exit with success or use the --preserve-status option to timeout to change exit code and have job considered as successful. Although if something is timing out it probably makes most sense to have the job marked as failed ?
Inside jenkins Execute shell you can wrap timeout with 'set -e' and 'set +e' so that the non-zero exit code will not be regarded as fail. Something like this would work for you:
set -e # no error if non-zero exit status
timeout <timeout> <terraform hanging command>
set +e
https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html
Appended to the first answer, you have a set alternative just piping the semicolon. Nasty workaround in Bash, but works.
timeout <timeout> <terraform hanging command> || :

Why does using csh -e option succeed on the command line, but fail in a jenkins execute shell?

I am using jenkins to build a bunch of legacy code. The legacy code comes with some complex build scripts, written in csh.
The build scripts do not check for or exit on errors. The user is expected to scan the output for error messages. However, this does not work well with Jenkins.
I am executing the csh build scripts in a jenkins "shell execution" build step. For example:
export PATH=`pwd`/ALL/bin:/usr/local/bin:/usr/bin:/bin:$PATH
cd ATLb2.00/expt_02.0
csh 020.com
When I run this from the command line, I can also use the -e option:
csh -e 020.com
In this case, as I expect, the script is run, but when the first error is encountered, the script stops and returns a non-zero code. However, when I try this in Jenkins, the build fails as soon as it gets to the csh -e command, without executing any of the script.
The error I get in Jenkins is:
+ csh -e 020.com
Build step 'Execute shell' marked build as failure
On the command line, the script is run and I see all kinds of output, until something fails, and then the script exits. On Jenkins the script seems to fail without even running. There is no output, and even scripts with no failures will not run for me under jenkins with the -e option.
What's up?
I recommend that you specify csh on a more global level and then execute the commands in a Jenkins build step.
If you want to use csh for all jobs, you can set the default shell using Jenkins > Manage Jenkins > shell executable.
If you want to use csh for only a particular job, begin the Execute shell build step with a shebang, such as:
#!/usr/bin/tcsh -e -x
command1
command2
...
Since I have tested only tcsh, that is what I use in the example.
Beware that a space is not allowed after the #!:
#! /usr/bin/tcsh # Wrong
This will give the error,
java.io.IOException: Cannot run program ""
I tested the above on Jenkins 1.625.3

Why Jenkins is failing after executing an external windows command line?

I have to create a job in my local Jenkins installation where it executes a SonarQube analysis and then calls a command-line program which searches for duplicated lines of code. However, when I execute the latter command (cpd), it runs okay since it outputs correctly in a external file, but Jenkins still points out it as an error like this:
E:\BASE_DIR>cpd --minimum-tokens 100 --files "E:\BASE_DIR\Project_Folder" --language java 1>>"E:\BASE_DIR\Project_Folder\CPD_CLIG.txt"
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I've tried to create another script which calls that command but I've got the same result.
Any suggestions on how to handle this? Any work-around would be very helpful.
Simple and short answer to your question is
Please add following line into your "Execute shell" Build step.
"#!/bin/sh"
Now let me explain you the reason why we require this line for "Execute Shell" build job.
By default Jenkins take "/bin/sh -xe" and this means -x will print each and every command.And the other option -e, which causes shell to stop running a script immediately when any command exits with non-zero(when any command fails) exit code.
So by adding the "#!/bin/sh" will allow you to execute with no option.
please refer : How/When does Execute Shell mark a build as failure in Jenkins?
I didn't find a proper solution for my issue but I realized that I can use a Jenkins plugin for static code analysis (DRY Plugin and PMD Plugin) to adress my problem.
Thanks you all.

YSlow Phantomjs and Jenkins jobs failing, but analysis successful

I'm going through the tutorial on YSlow and Phantom js in Jenkins here: http://yslow.org/phantomjs/
Everything appears to be working great except the Jenkins builds are failing. I think this is due to the violations that YSlow is finding (6 for the particular site I am measuring). I'd rather have the build be successful (or unstable) vs. failed though
Is that possible with this or will I have to resort to something like the postgroovy or text finder plugin?
This is the console output:
phantomjs.exe yslow.js -i grade -t 50 --format junit http://www.somesite.com 1>yslow.xml
D:\Apps\Jenkins\workspace\YSlow_Test>exit 6
Build step 'Execute Windows batch command' marked build as failure
Thanks
Any non-zero exit code at the end of your Execute Windows batch command build step will result in build step being marked as failure.
To have the build step marked as success, you need an exit code of 0. I don't know anything about "yslow" or "phantomjs" and why they are giving you exit code of non-zero, but from "batch" side of things, you need only write exit 0 at the end of your build step if you want to overwrite the exit code of your phantomjs command.
You can then use Text Finder plugin to parse the console log and mark build as unstable when certain conditions are met.
Reading over this answer, Configuring yslow on Jenkins looks like you need TAP plugin to have the functionality of unit testing marking the build as unstable automatically

Resources