We are repeatedly seeing resource quota limitation issues in logs and Task jobs fail on the SCDF running on Kubernetes. Problem is, there are so many pods in "running" status even after they completed. I understand, SCDF does not delete the pods and it is developer's responsibility to cleanup.
Even when I run the Task Execution Cleanup from SCDF dashboard UI, it only cleans up the execution logs and task form UI but the pods created by that task still remain. Is this expected ? Shouldn't Task Execution Cleanup also delete the pods ? We are using Spring-Cloud-Dataflow-Server 2.4.2 Release.
Is there a way to cleanup the pods right after the execution is complete ? Any best practices here ?
Method - 1
You can clean up task executions by using restful api provided by spring-cloud-dataflow.
Request Structure
DELETE /tasks/executions/{ids}?action=CLEANUP,REMOVE_DATA HTTP/1.1
Host: localhost:9393
Fire DELETE request.
http://<stream-url>/tasks/executions/<task-ids-seperated-by-coma>?action=CLEANUP,REMOVE_DATA
eg: http://localhost:9393/tasks/executions/1,2?action=CLEANUP,REMOVE_DATA
Delete Task Execution
Note: Above api will clean up resources that were used to deploy tasks and delete the data associated with task executions from the underlying persistence store.
CLEANUP : clean up the resources
REMOVE_DATA : remove data from persistence store.
You can either pass both actions or single action depends on your use-case.
Method - 2
Using spring cloud dataflow shell.
Enter into the spring-cloud-dataflow shell and execute below command.
task execution cleanup --id <task-id>
eg: task execution cleanup --id 1
Cleanup task execution from spring-cloud-dataflow shell
Other option (Applicable for kubernetes platform only)
If you wan't to delete all completed pods then you can delete using kubectl tool.
kubectl delete pod --field-selector=status.phase==Succeeded -l role=spring-app -n <namespace-where-tasks-launched>
If you wan't to delete all pods with Error status then execute below command
kubectl delete pod --field-selector=status.phase==Failed -l role=spring-app -n <namespace-where-tasks-launched>
Related
I have mutiple stages in docker. I am using both rolling update and deployement. currently if nothing is deployed I have to do deploy manually and every time I update rolling update will be run by itself.
I want to automate everything. I want to write some condition to check if
there is existing replication controller then run rolling update else run deploy. I tried to put if statements in scripts in both deploy or rollout update.
- >
if rancher --debug kubectl rollout status deployment $CI_PROJECT_NAME; then
exit 0
fi
unfortunately I got error:
ERROR: Job failed (system failure): secrets
"runner-yx1bv22m-project-122-concurrent-0vrbdb" is forbidden: exceeded
quota: default-467s2, requested: secrets=1, used: secrets=20, limited:
secrets=20
from what I understand I can't run both jobs at once. what Is the right way to achieve either deployment or rolling update depending on current deployment status?
Basically I got rid of second job. I do everything in single job now.
script:
other scripts
- CHECK_DEPLOYMENT=$(rancher --debug kubectl --insecure-skip-tls-verify get deployments -n $CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG | grep $CI_PROJECT_NAME | awk '{print $1}')
- >
if [ "$CHECK_DEPLOYMENT" == "$CI_PROJECT_NAME" ]; then
do rollup script
else
do deploy script
fi
both deployment or rollup is done by adding single if statement. depending on deployment it will do either roll up or it will deploy.
I have some doubts about the correct configuration of Jenkins to ensure the continuous integration of a Drupal project but I arrive at some contradictions.
Let me explain: the deployment, after all, consists in executing:
cd / path / to / web / root
pull from git
drush config:import
drush cache:rebuild
The tests are launched with the command
../vendor/bin/phpunit --verbose --log-junit ../tests_output/phpunit.xml -c ../phpunit.xml
The contradiction is that I do not understand when to run the tests.
Before the pull does not make sense because the last changes are missing, after the pull if any test goes wrong I should be able to restore the situation before the pull (but I'm not sure it's a safe action).
I'm trying to run the tests directly in the workspace of jenkins and to do this I also created a separate database, but at the moment I get the error:
Drupal\Tests\field_example\Functional\TextWidgetTest::testSingleValueField
Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Log in" not found
What could be the best strategy to follow?
So, your order seems ok - pull first then run tests.
However, you can have 2 Jenkins jobs. First runs, your tasks. 2nd runs ONLY if your first job completes without failure.
There are ways to get exit status from scripts - see following plugins/notes about that.
How to mark Jenkins builds as SUCCESS only on specific error exit values (other than 0)?
How to mark a build unstable in Jenkins when running shell scripts
I am building Jenkins for Test / QA automation scripts, lets name it TEST_JOB. For application, I have application source code Jenkins build, name it DEV_JOB.
My scenario is when DEV_JOB completes execution (successfully), execute TEST_JOB immediately. I am aware about setting up project upstream / downstream [ Build after other projects are built ] to accomplish this task. But here, Problem is DEV_JOB is on different server than TEST_JOB. Due to which, TEST_JOB fails to recognize DEV_JOB.
Now, how would I achieve this scenario?
You can use Jenkins API for remote trigger of Job.
Say you have job on DEV_JOB on JENKINS_1, add a penultimate step(or upstream/downstream project having only this step) which invokes TEST_JOB using remote API call of JENKINS_2 server.
Example command would be
$(curl --user "username:password" "http://JENKINS_2/job/TEST_JOB/buildWithParameters?SOMEPARAMETER=$SOMEPARAMETER")
username:password is a valid user on JENKINS_2.
Avoid using your own account here but rather a 'build trigger' account that only has permissions to start those jobs.
I'm in the process of migrating Jenkins from one server to another. I've no issues with the migration process.
But sooner I start my new server the scheduled jobs start executing, which is proving to be dangerous. I need to make sure that everything is in place before activating the new server.
Is there any way to deter any of the jobs from executing while the new server is active?
execute an '/script':
Jenkins.instance.getAllItems(org.jenkinsci.plugins.workflow.job.WorkflowJob.class).each {i -> i.setDisabled(true); i.save() }
Jenkins.instance.getAllItems(hudson.model.AbstractProject.class).each {i -> i.setDisabled(true); i.save() }
Not my idea, from jenkins wiki
Setup a post-initialization script that puts Jenkins into quiet mode right after startup.
https://wiki.jenkins.io/display/JENKINS/Post-initialization+script
Try using https://wiki.jenkins.io/display/JENKINS/Exclusive+Execution+Plugin. You can keep jenkins in shutdown or Quiet mode for some time till your new instance is ready to function.
Use the Jenkins CLI
To prevent any jobs from being run, use quiet-down:
java -jar jenkins-cli.jar -s http://localhost:9090 -auth user:token quiet-down
To re-enable job scheduling:
java -jar jenkins-cli.jar -s http://localhost:9090 -auth user:token cancel-quiet-down
Scheduled jobs will be added to the queue during the quiet-down time, and will be run after canceling the quiet-down. If that's not what you want, you may use clear-queue before canceling the quiet-down.
There is a little downside: in the GUI, Jenkins will announce that it is preparing for shutdown, which wouldn't be true in this case. I find that acceptable, because we use it during backup at night when no one will read the announcement anyway. However, another option would be to take nodes offline, then online again using offline-node and online-node.
Quick Setup
Only if you haven't set up Jenkins CLI already:
You can obtain the Jenkins CLI from your Jenkins server by downloading it from <your_jenkins_url>/jnlpJars/jenkins-cli.jar
Instead of using your actual password to authenticate, obtain a token from <your_jenkins_url>/me/configure
For more details, refer to the Jenkins Handbook: Jenkins CLI
Referencie: https://xanderx.com/post/cancel-all-queued-jenkins-jobs/
Run this in Manage Jenkins > Script Console:
Jenkins.instance.queue.clear()
I am running a background process through a script , this script is invoked when Jenkin starts building. However, the jenkins build gets stuck and on looking at the console it seems it is running the process and waits for it to complete.
This process will never complete, consider this as a server listening to its client.Every build I trigger kills the server process and restarts the process, so I am perfectly handling that scenario.
Is there any way , I can build jenkins successfully?
The exact details depend on your operating system (which you did not tell), but the Jenkins wiki has a page about this: https://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build
There is a trick you can do in order you free a Jenkins thread.
What you can do is to execute a bash script through a ssh connection and send it to the background while saving the pid of the process somewhere so you can make checks further.
The format of the command would be:
ssh -n _hostname_ "_commands_ & echo \$! > \"_path_to_pid_file_\"" &
Example with a never-ending program:
ssh -n myhost.domain.com "tail -f /var/log/my.log & echo \$! > \"$WORKSPACE/pid\"" &
This example will spawn the tail process listening forever for new changes in the /var/log/my.log file and store its pid in the $WORKSPACE/pid file.
When executed from a Jenkins job the ssh process will exit immediately while the commands sent to the background will remain in execution in the specified host.
I do this in order to maintain always one of the services I run in my build farm in-sync with the latest code modification of it in the repository.
Just have a job that ssh' into the target machine and then kill the process, update the service and re-launches it.
This could be a bit cumbersome but it works great!