I have a Jenkins project in which i run a sonarqube analysis in a windows OS.
In jenkins , I created a batch step more or less like this:
mycommand test --machine --coverage > tests.output
sonar-scanner
mycommand is a 3rd party plugin which i can't modify , and, based on the content of the project , this step can fail and I want the jenkins queue to go on with the other command .
Now if mycommand return an error jenkins stop.
How can I achieve this?
The magic here is in the return code from your command. Without knowing what it is and what it does, the important thing is that if any command returns non-zero exit code Jenkins will see the command as failed. To prevent any errors from being registered by Jenkins add exit 0 to the command.
For example
bat """
mycommand test --machine --coverage > tests.output || exit /b 0
"""
bat """
sonar-scanner
"""
Should work for this.
Alternatively you can also chain the commands together with & so Jenkins doesn't check the exit codes before everything is completed, but then the second command will also always show as passed regardless of its real status. Also it can become difficult to read with lengthy command chains.
bat """
mycommand test --machine --coverage > tests.output & sonar-scanner & exit /b 0
"""
Related
I want to be able to read the exit code of the build script, from my post-build script.
Is there a way to do that in Jenkins configuration?
it will allow me to check for a matching string (in this case 'hella success rn') but it won't let me see the exit code of the build script?
You may use #!/bin/sh at the start of your Execute shell block as shown below:
Using #!/bin/sh without the default options that Jenkins uses i.e., #!/bin/sh -xe will prevent Jenkins from marking your Execute shell as failure. Doing this will help you run the subsequent command of finding the return status of your script.
-e Exit immediately if a command exits with a non-zero status.
Ref: http://linuxcommand.org/lc3_man_pages/seth.html
Based on the return status (script_status=$?) of your script, print something that you can search later in your post-build step.
Then, in the Post-build Actions, search that printed text to decide your post-build action as shown below:
Output:
My query is related to Jenkins server.
I have made one API to hit the Jenkins server where Jenkins starts test suites.
My question is: can Jenkins server return 0 if any test case fail, and 1 otherwise?
The API URL is in the form
JENKINS_URL/job/Encore_Automation/build?token=TOKEN_NAME
By looking at Build Triggers / Trigger builds remotely (e.g., from scripts) it seems like this option only supports queuing a project and it does not let you retrieve results.
Jenkins REST API
After build has been triggered from REST API call, you could start making consecutive REST API calls to check it status.
Jenkins CLI
However Jenkins offers a jenkins-cli tool which let you not only to trigger the build but also to wait until its completion:
java -jar jenkins-cli.jar -s http://localhost:8080/ build JOB [-c] [-f] [-p] [-r N] [-s] [-v] [-w]
Starts a build, and optionally waits for a completion.
Aside from general scripting use, this command can be
used to invoke another job from within a build of one job.
With the -s option, this command changes the exit code based on
the outcome of the build (exit code 0 indicates a success)
and interrupting the command will interrupt the job.
With the -f option, this command changes the exit code based on
the outcome of the build (exit code 0 indicates a success)
however, unlike -s, interrupting the command will not interrupt
the job (exit code 125 indicates the command was interrupted).
With the -c option, a build will only run if there has been
an SCM change.
JOB : Name of the job to build
-c : Check for SCM changes before starting the build, and if there's no
change, exit without doing a build
-f : Follow the build progress. Like -s only interrupts are not passed
through to the build.
-p : Specify the build parameters in the key=value format.
-s : Wait until the completion/abortion of the command. Interrupts are passed
through to the build.
-v : Prints out the console output of the build. Use with -s
-w : Wait until the start of the command
Due some problems with the hanging of a python process (yandex-tank) during the build process in Jenkins (after which the build could not stop) i need to stop this problematic process with some additional kill command with timeout or using timeout command itself:
timeout $TIMEOUT yandex-tank-jmeter -i -o "jmeter.jmx=$WORKSPACE/$TEST_PLAN"
timeout sends default (15) kill signal, but after that the build goes to status FAILED.
Is there any workaround or special kill signal to make builds successful ?
Have you tried manual exit code overriding?
timeout $TIMEOUT yandex-tank-jmeter -i -o "jmeter.jmx=$WORKSPACE/$TEST_PLAN"; RES=$?
//If the command timed out, then RES equals 124.
...
//at the end of job scenario:
if [ $RES -eq 124 ]; then RES=0;
fi
exit $RES
According to the Jenkins documentation for the "Execute shell" step:
By default, the shell will be invoked with the "-ex" option.
Therefore, Jenkins places all shell code into a shell script file, in the temp directory, something like /tmp/sh/jenkins45723947385985.sh and then executes it as follows:
/bin/sh -xe /tmp/sh/jenkins45723947385985.sh
This can be seen in the console output of the job.
The e option in -xe means that the shell will exit as soon as it has an error. To change this behaviour add a custom shebang line to the start of the Jenkins shell script such as
#!/bin/sh -x
Jenkins will no longer terminate as soon as an error occurs.
Just wanted to explore pytest and integrating it into Jenkins. My sample pytest test cases are
def a(x):
return x+1
def test_answer():
assert a(2) == 3
def test_answer2():
assert a(0) == 2
I then generated a standalone pytest script which I run in Jenkins, generating an xml to be parsed for results.
As test_answer2 fails, the Jenkins job also fails. I'm assuming this is because the exit code returned is non-zero. How would I go around this, i.e the Jenkins job doesn't even if 1 or more tests do indeed fail. Thanks
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Windows:
<your test execution calls>
exit 0
Linux:
set +e
<your test execution calls>
set -e
This will ignore the error if at all it is called with in the batch scripts and the Jenkins will show status as successful.
In addition to already posted answers:
You also can mark your test as xfail, what means you know it will fail, just like skipping:
#pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
...
more about skipping you can find in pytest documentation
and on Jenkins side you also can manipulate of state of your build via simply try/catch statement:
try {
bat "python -m pytest ..."
} catch (pytestError) {
// rewrite state of you build as you want, Success or Failed
// currentBuild.result = "FAILED"
currentBuild.result = "SUCCESS" // your case
println pytestError
}
But be aware, it will mark whole build each time as success for that step of pytest run. Best practice just to skip tests via #pytest.mark.skip as described above.
If you are calling this test execution in a batch file or shell script or directly using the command execution in Jenkins. You can follow the below way:
Below code is NOT Working
Linux:
set +e
set -e
We use Jenkins running on a Windows system so our tests are listed in the Jenkins "Execute Windows Batch command" section.
I was able to solve this by separating the tests that might have failures with a single & (rather than &&). For example:
"C:\Program Files\Git\bin\sh.exe" -c python -m venv env && pip3 install -r requirements.txt && py.test --tap-files test_that_may_fail.py & py.test --tap-files next_test.py & py.test
Since we use pytest, any failures are flagged in python with an assert. If you use the &&, this will cause Jenkins job to abort and not run the other tests.
I have a lots of tests on several server. I use Jenkins to manage all of them.
On one server (Slave Windows), when I launch a test in cmd, I got something like :
c:/tests/cucumber --tag #dev -p ie
...
1 scenarios (1 passed)
12 steps (12 passed)
0m31.761s
echo %errorlevel%
0
No error in the tests, and cucumber seems good.
When jenkins launch exactly the same tests, I get :
c:/jenkins_folder/cucumber --tag #dev -p ie
...
1 scenarios (1 passed)
12 steps (12 passed)
0m28.453s
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
The test passed, but is marked failed by jenkins.
The command "echo %errorlevel%" is aborted : the job fail before this point.
The same job played on another slave work.
Same problem with all profile and all tags.
Same problem when I replace profile by real value
I don't use the --strict flag
Jenkins, plugins : all up-to-date
Code of the windows batch :
cd /test8folder
cucumber --tag #dev -p ie
echo %errorlevel%
What did I missed ?
There doesn't seem to be anything wrong with the configuration.
The fact that you get Build step 'Execute Windows batch command' marked build as failure after your cucumber command without actually seeing the echo %errorlevel% executed only reaffirms that there was an error in cucumber (more on that later).
However, in Execute Windows Batch Command, even a command in error would not exit the batch script (unlike Jenkins's default Execute Shell implementation). You should be seeing at least some exit code, 0, 1, or anything.
The only time this would happen is if something within your buildstep executed exit /b [exit_code_num]. I don't know "cucumber", but if that is in-fact a cucubmer.bat and inside there is an exit /b statement, this is what's causing it drop out of the buildstep without continuing.
Solution
You can use call cucumber [whateverparams] so that even when it quits with exit /b, the control will return to the calling process, the Execute Windows Batch command script.
Try that first. And you will probably see that your echo %errorlevel% will probably return a non-zero value when executed under Jenkins, but at least you will see it now.
Now, as for why it succeeds on command prompt, but fails within Jenkins, there could be a lot of reasons, the most common one being environment variables and paths. We can tackle that later once we actually see the exit code of cucumber. You also said it worked on another node: even more reason to believe this is an environment issue, maybe a non-existent folder...
Edit:
The reason that even "successful" test execution exits the calling script is because exit /b 0 would still quit the calling script, even though cucumber exited with "success" 0