I have a Jenkins pipeline where I have the following code (jenkinsfile):
stage('My stage') {
options {
timeout(time: 5, unit: "MINUTES")
}
steps {
script {
./run_script.sh
}
}
}
}
With this code I was expecting that If the run_script.sh would take more than 5 minutes, the job would abort.
But this is not the case (even with jenkins telling me "Sending interrupt signal to process").. After running the job I get this:
...
17:09:40 Timeout set to expire in 5 min 0 sec
[Pipeline] {
[Pipeline] script
[Pipeline] {
17:09:40 ./run_script.sh '
17:14:40 Cancelling nested steps due to timeout
17:14:40 Sending interrupt signal to process
17:14:48 ...
17:14:48 + JOB_NAME=...
17:14:48 + BUILD_URL=...
17:14:48 + BUILD_ID=777
17:14:48 + BUILD_RESULT=SUCCESS
The job gets executed and I can see the green success status in jenkins.
Why is this happening and how can I make it so when a timeout occurs the job gets aborted?
EDIT:
Not sure if this is relevant or not but the run_script.sh is a script which connects to a remote server using ssh and runs some docker commands (docker run...). Running this script using timeout gives me the following msgs:
...
12:07:22 Cancelling nested steps due to timeout
12:07:22 Sending interrupt signal to process........
.........
Finished: SUCCESS
As you can see the status stays success instead of aborted!!!
I need to capture this "timeout" so I can send notifications to users. One way would be to capture the aborted status but if the status is success how can i do it?
The run_script.sh looks like this:
#!/bin/bash
ssh user#server.com "
ssh -p 30 xdy#localhost ' docker run --rm --name cont1 -e FTP_USER=$User -e FTP_PASSWD=$Pass -v /home/logs:/usr/myapp -w /usr/myapp server2.com:3000/mylog:1.1 python script.py' "
script.py is a simple script to copy some files from a server to another.
Related
I have a site in production. And I have a simple playwright test that browses to the site and does some basic checks to make sure that it's up.
I'd like to have this job running in Jenkins every 5 minutes, and if the tests fail I want to run a script that will restart the production server. If the tests pass, I don't want to do anything.
What's the easiest way of doing this?
I have the MultiJob plugin that I thought I could use, and have the restart triggered on the failed test step, but it doesn't seem to have the ability to trigger specifically on fail.
Something like the following will do the Job for you. I'm assuming you have a second Job that will take care of the restart.
pipeline {
agent any
triggers{
cron('*/5 * * * *')
}
stages {
stage("Run the Test") {
steps{
echo "Running the Test"
// I'm returning exit code 1 so jenkins will think this failed
sh '''
echo "RUN SOMETHING"
exit 1
'''
}
}
}
post {
success {
echo "Success: Do nothing"
}
failure {
echo 'I failed :(, Execute restart Job'
// Executing the restart Job.
build job: 'RestartJob'
}
}
}
I'm running a basic pipeline that executes pylint on my repository code.
My Jenkins runs on Debian etch, the Jenkins version is 2.231.
At the end of the pipeline, I'm getting the following error :
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 24
Finished: FAILURE
As this page https://wiki.jenkins.io/display/JENKINS/Job+Exit+Status explains, the error code 24 refers to a "too many open files" error.
If I remove the pylint part of the job, the pipelines executes smoothly.
Il tried to configure the limit in /etc/security/limits.conf
jenkins soft nofile 64000
jenkins hard nofile 64000
and in the Jenkins config file /etc/default/jenkins :
MAXOPENFILES=64000
If I put "ulimit -n" in the pipeline, the configured value is displayed but it has no effect on the result that remains : ERROR: script returned exit code 24.
The problem comes from pylint that doesn't return a 0 code even if it ran successfully.
The solution is to use the --exit-zero option when running pylint.
pylint --exit-zero src
Jenkins with pylint gives build failure
In my Jenkins pipeline, I have the following Jenkins step constructed using Jenkins scripted syntax. Its purpose is to run a docker container, which runs some integration tests. I currently spawn 11 of these steps in parallel.
node('tester') {
stage("Test") {
withEnv([
"USERNAME=some_user",
"UID=1000",
"GID=1000"
]) {
dir("some_directory") {
// ... some steps here
retry(1) {
try {
timeout(time: 15, unit: 'MINUTES') {
retry(1) {
sh "docker-compose --project-name tester_prod9 -f docker-compose.e2e.yml -- up --build --renew-anon-volumes --exit-code-from cypress --timeout 600"
}
}
} catch (err) {
error "Timeout!"
}
}
}
}
}
}
As you can see, I want my docker-compose command to retry once if the docker-compose command fails. However, if this process (including a potential retry) exceeds 15 minutes, I want a timeout error to be thrown. If a timeout error is thrown, I would all of this to be retried once more.
However, the above desired steps are not happening.
Currently, when my docker-compose command fails on the first run, a "Timeout!" error is thrown and the whole step ends.
As you can see below, the docker-compose command only runs for 8 minutes so a timeout error shouldn't be thrown. Also, the docker-compose command is never retried by Jenkins.
How can I fix my Jenkins step in order to achieve my desired behaviour?
My Jenkins version is 2.270.
Edit:
Just to be clear, I expect the docker-compose command to fail because there is a failing test inside the Docker container. However, Jenkins isn't retrying the failed docker-compose command as expected.
I have not found anywhere any information about retry should be > 1. Official documentation says:
retry: Retry the body up to N times
Retry the block (up to N times) if any exception happens during its body execution.
If an exception happens on the final attempt then it will lead to aborting the build
(unless it is caught and processed somehow). User aborts of the build are not caught.
But it looks that in order to trigger retries it should be set to more than 1
I have a requirement to implement distributed performance testing where I have a chance of launching multiple slave node parallelly when user count is high. Hence I suppose to launch master and slave nodes.
I have tried all the way to start jmeter-server in the background since it has to keep on running in the slave node to receive the incoming request.
But still, I am unable to start it in the background.
node(performance) {
properties([disableConcurrentBuilds()])
stage('Setup') {
cleanAndInstall()
checkout()
}
max_instances_to_boot = 1
for (val in 1..max_instances_to_boot) {
def instance_id = val
node_builder[instance_id] = {
timestamps {
node(node_label) {
stage('Node -> ' + instance_id + ' Launch') {
def ipAddr = ''
script {
ipAddr = sh(script: 'curl http://xxx.xxx.xxx.xxx/latest/meta-data/local-ipv4', returnStdout: true)
node_ipAddr.add(ipAddr)
}
cleanAndInstall()
checkout()
println "Node IP Address:"+node_ipAddr
dir('apache-jmeter/bin') {
exec_cmd = "nohup sh jmeter-server -Jserver.rmi.ssl.disable=true -Djava.rmi.server.hostname=$ipAddr > ${env.WORKSPACE}/jmeter-server-nohup.out &"
println 'Server Execution Command: ' + exec_cmd
sh exec_cmd
}
sleep time: 1, unit: 'MINUTES'
sh """#!/bin/bash
echo "============ jmeter-server.log ============"
cat jmeter-server.log
echo "============ nohup.log ============"
cat jmeter-server-nohup.out
"""
}
}
}
}
}
parallel node_builder
stage('Execution') {
exec_cmd = "apache-jmeter/bin/jmeter -n -t /home/jenkins/workspace/release-performance-tests/test_plans/delights/fd_regression_delight.jmx -e -o /home/jenkins/workspace/release-performance-tests/Performance-Report -l /home/jenkins/workspace/release-performance-tests/JTL-FD-773.jtl -R xx.0.3.210 -Jserver.rmi.ssl.disable=true -Dclient.tries=3"
println 'Execution Command: ' + exec_cmd
sh exec_cmd
}
}
Getting following error
Error in rconfigure() method java.rmi.ConnectException: Connection refused to host: xx.0.3.210; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
We're unable to provide the answer without seeing the contents of your nohup.out file which is supposed to contain your script output.
Blind shot: by default JMeter uses secure communication between the master and the slaves so you need to have a Java Keystore to contain certificates necessary for the requests encryption. The script is create-rmi-keystore.sh and you need to launch and perform the configuration prior to starting the JMeter Slave.
If you don't need encrypted communication between master and slaves you can turn this feature off so you won't to create the keystore, it can be done either by adding the following command-line argument:
server.rmi.ssl.disable=true
like:
nohup jmeter-server -Jserver.rmi.ssl.disable=true &
or alternatively add the next line to user.properties file (lives in "bin" folder of your JMeter installation)
server.rmi.ssl.disable=true
More information:
Configuring JMeter
Apache JMeter Properties Customization Guide
Remote hosts and RMI configuration
This is resolve by adding inside the node stage.
JENKINS_NODE_COOKIE=dontKillMe nohup sh jmeter-server -Jserver.rmi.ssl.disable=true -Djava.rmi.server.hostname=xx.xx.xx.xxx > ${env.WORKSPACE}/jmeter-server-nohup.out &
I try to invoke the below salt script using Jenkins:
create_script:
file.managed:
- name: /tmp/broc/import_props.sh
- source: salt://projects/broc/jboss/files/import.sh.jinja
- template: jinja
Import_properties:
cmd.script:
- name: /tmp/broc/import.sh
- cwd: /tmp/broc`
The Jenkins console output is:
`ID: create_script
Function: file.managed
Name: /tmp/broc/import.sh
Result: True
Comment: File /tmp/broc/import.sh updated
Started: 11:31:13.736928
Duration: 166.319 ms
Changes:
----------
diff:
New file
mode:
0644
ID: Import_properties
Function: cmd.script
Name: /tmp/broc/import.sh
Result: False
Comment: Command '/tmp/broc/import.sh' run
Started: 11:31:13.903378
Duration: 399.825 ms
Changes:
----------
pid:
8292
retcode:
1`
And Jenkins build finished success:
`Succeeded: 21 (changed=22)
Failed: 1
Total states run: 22
Total run time: 30.338 s"}}]
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS`
My question is a saltstack job ID Import_properties Result is False. So a Jenkins build should also finish as FAILURE. In the above case the saltstack result is ignored and build finished SUCCESS. Is there a way the jenkins build be made FAILURE based on saltstack Result?
I see the below Jenkins pipeline:
`try{
saltCmd = "\"salt -E \"($target)\" \ state.apply projects.alip.process-server \
pillar=\'{\"region\": \"${Region}\",\"siteid\":\"${SiteID}\",\"dbuser\":\"${DBUSER}\",\"dbpass\":\"${DBPASS}\"}\' \""
result = salt authtype: 'pam',
clientInterface: local(
arguments: saltCmd,
blockbuild: true,
function: 'cmd.run',
target: "$my_salttarget",
saveFile: true,
targettype: 'glob'),
credentialsId: "$my_saltcred",
servername: "$my_saltserver"
}
}catch(e){
result = e.toString()
currentBuild.result = 'FAILURE'
}finally{
echo result.replace("\\n",'\n')
}
}`
I am new to Jenkins pipeline script, can you help suggest inputs for adding a post build steps under finally to parse the Jenkins console output, identify a string and if it matches mark the build failure. This is similar to a text finder plugin except that we write a pipeline script.
Even if your state failed, Jenkins believe the salt command it evoked still ran successfully.
Jenkins is not able to detect the errors inside salt itself.
It can only tell if salt runs successfully or not.
This is the same as when you ran salt command line. Even if the state fail, the shell command salt still returns 0.
Your issue here is (as said by others) due to jenkins using the return code if the salt command itself and NOT the return code of the action taken by the state you applied.
My 2 cents here is the
--retcode-passthrough
option you can pass to your salt command.
This option allows the salt command return code to match the action taken by the state.
Simply put, if anything fails in a state then the salt command will return a failure return code.
Official doc here