Due some problems with the hanging of a python process (yandex-tank) during the build process in Jenkins (after which the build could not stop) i need to stop this problematic process with some additional kill command with timeout or using timeout command itself:
timeout $TIMEOUT yandex-tank-jmeter -i -o "jmeter.jmx=$WORKSPACE/$TEST_PLAN"
timeout sends default (15) kill signal, but after that the build goes to status FAILED.
Is there any workaround or special kill signal to make builds successful ?
Have you tried manual exit code overriding?
timeout $TIMEOUT yandex-tank-jmeter -i -o "jmeter.jmx=$WORKSPACE/$TEST_PLAN"; RES=$?
//If the command timed out, then RES equals 124.
...
//at the end of job scenario:
if [ $RES -eq 124 ]; then RES=0;
fi
exit $RES
According to the Jenkins documentation for the "Execute shell" step:
By default, the shell will be invoked with the "-ex" option.
Therefore, Jenkins places all shell code into a shell script file, in the temp directory, something like /tmp/sh/jenkins45723947385985.sh and then executes it as follows:
/bin/sh -xe /tmp/sh/jenkins45723947385985.sh
This can be seen in the console output of the job.
The e option in -xe means that the shell will exit as soon as it has an error. To change this behaviour add a custom shebang line to the start of the Jenkins shell script such as
#!/bin/sh -x
Jenkins will no longer terminate as soon as an error occurs.
Related
I have a Jenkins project in which i run a sonarqube analysis in a windows OS.
In jenkins , I created a batch step more or less like this:
mycommand test --machine --coverage > tests.output
sonar-scanner
mycommand is a 3rd party plugin which i can't modify , and, based on the content of the project , this step can fail and I want the jenkins queue to go on with the other command .
Now if mycommand return an error jenkins stop.
How can I achieve this?
The magic here is in the return code from your command. Without knowing what it is and what it does, the important thing is that if any command returns non-zero exit code Jenkins will see the command as failed. To prevent any errors from being registered by Jenkins add exit 0 to the command.
For example
bat """
mycommand test --machine --coverage > tests.output || exit /b 0
"""
bat """
sonar-scanner
"""
Should work for this.
Alternatively you can also chain the commands together with & so Jenkins doesn't check the exit codes before everything is completed, but then the second command will also always show as passed regardless of its real status. Also it can become difficult to read with lengthy command chains.
bat """
mycommand test --machine --coverage > tests.output & sonar-scanner & exit /b 0
"""
In a job where i call terraform apply, after the deploy the shell is stuck and the job doesn't end so i have to abort it. Is it possible to mark the aborted job as success or there is another way to end this job without force?
On the job summary/artifacts/changes/triggers/results page you can "add a description". This is a good place to summarize results and explain what happened after job result is reviewed. The first part of description also appears in Build History.
Answer:
In shell scripts (or jenkins job script) you can use bash timeout command to protect any command that might get stuck. 'timeout [<option>] <duration> <command>'. e.g.
$ timeout 2s sleep 4
$ echo $?
124
You can check this exit value in script and exit with success or use the --preserve-status option to timeout to change exit code and have job considered as successful. Although if something is timing out it probably makes most sense to have the job marked as failed ?
Inside jenkins Execute shell you can wrap timeout with 'set -e' and 'set +e' so that the non-zero exit code will not be regarded as fail. Something like this would work for you:
set -e # no error if non-zero exit status
timeout <timeout> <terraform hanging command>
set +e
https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html
Appended to the first answer, you have a set alternative just piping the semicolon. Nasty workaround in Bash, but works.
timeout <timeout> <terraform hanging command> || :
My query is related to Jenkins server.
I have made one API to hit the Jenkins server where Jenkins starts test suites.
My question is: can Jenkins server return 0 if any test case fail, and 1 otherwise?
The API URL is in the form
JENKINS_URL/job/Encore_Automation/build?token=TOKEN_NAME
By looking at Build Triggers / Trigger builds remotely (e.g., from scripts) it seems like this option only supports queuing a project and it does not let you retrieve results.
Jenkins REST API
After build has been triggered from REST API call, you could start making consecutive REST API calls to check it status.
Jenkins CLI
However Jenkins offers a jenkins-cli tool which let you not only to trigger the build but also to wait until its completion:
java -jar jenkins-cli.jar -s http://localhost:8080/ build JOB [-c] [-f] [-p] [-r N] [-s] [-v] [-w]
Starts a build, and optionally waits for a completion.
Aside from general scripting use, this command can be
used to invoke another job from within a build of one job.
With the -s option, this command changes the exit code based on
the outcome of the build (exit code 0 indicates a success)
and interrupting the command will interrupt the job.
With the -f option, this command changes the exit code based on
the outcome of the build (exit code 0 indicates a success)
however, unlike -s, interrupting the command will not interrupt
the job (exit code 125 indicates the command was interrupted).
With the -c option, a build will only run if there has been
an SCM change.
JOB : Name of the job to build
-c : Check for SCM changes before starting the build, and if there's no
change, exit without doing a build
-f : Follow the build progress. Like -s only interrupts are not passed
through to the build.
-p : Specify the build parameters in the key=value format.
-s : Wait until the completion/abortion of the command. Interrupts are passed
through to the build.
-v : Prints out the console output of the build. Use with -s
-w : Wait until the start of the command
I am using jenkins to build a bunch of legacy code. The legacy code comes with some complex build scripts, written in csh.
The build scripts do not check for or exit on errors. The user is expected to scan the output for error messages. However, this does not work well with Jenkins.
I am executing the csh build scripts in a jenkins "shell execution" build step. For example:
export PATH=`pwd`/ALL/bin:/usr/local/bin:/usr/bin:/bin:$PATH
cd ATLb2.00/expt_02.0
csh 020.com
When I run this from the command line, I can also use the -e option:
csh -e 020.com
In this case, as I expect, the script is run, but when the first error is encountered, the script stops and returns a non-zero code. However, when I try this in Jenkins, the build fails as soon as it gets to the csh -e command, without executing any of the script.
The error I get in Jenkins is:
+ csh -e 020.com
Build step 'Execute shell' marked build as failure
On the command line, the script is run and I see all kinds of output, until something fails, and then the script exits. On Jenkins the script seems to fail without even running. There is no output, and even scripts with no failures will not run for me under jenkins with the -e option.
What's up?
I recommend that you specify csh on a more global level and then execute the commands in a Jenkins build step.
If you want to use csh for all jobs, you can set the default shell using Jenkins > Manage Jenkins > shell executable.
If you want to use csh for only a particular job, begin the Execute shell build step with a shebang, such as:
#!/usr/bin/tcsh -e -x
command1
command2
...
Since I have tested only tcsh, that is what I use in the example.
Beware that a space is not allowed after the #!:
#! /usr/bin/tcsh # Wrong
This will give the error,
java.io.IOException: Cannot run program ""
I tested the above on Jenkins 1.625.3
I get the following console output while running a job on Jenkins:
Building in workspace /home/admin/.jenkins/jobs/ramdisk/workspace
[SSH] executing pre build script:
./build_script.sh
[SSH] exit-status: 127
[SSH] executing post build script:
[SSH] exit-status: 0
Finished: SUCCESS
When I run the above script manually on a terminal, it is fine.
Can you please guide what could be the issue here?
First of all, when you run script manually on the terminal, you could be getting exit status 127 as well, but since you are not checking the exit status on terminal, you are not noticing it.
Jenkins assumes that only exit code of 0 means success. Any other exit code means some failure. If your build_script.sh uses exit codes in non-standard way, it will make Jenkins think that it had failed.
Do the following directly on the machine:
./build_script.sh
echo $?
Make sure you do the echo command immediately after the script command, without doing anything else. This will print the exit code on the command line. Answer here what you get.
Second, open the build_script.sh in a text editor (vim, nano, whatever you use) and search for exit 127. If you find it, see what block does it exit from. If you got trouble reading the script, paste the content of the script (or just the block where you found exit 127) in here, and we could tell you why the script thinks it needs to exit.