I have a shell script in my git which downloads huge android workspace (takes 90 minutes) & performs some operations & refines it & then I trigger some checks on another tool(via same shell script).
So until now I have configured the jenkins where it checkouts the shell script & runs it.
This shell script takes 90 minutes to download such a huge repo IN TO THE SAME WORKSPACE & then triggers some check on other tool(via its api).
But problem is if i just do minor improvements to the shell script & re run the job the entire workspace with previously refined sources gets wiped out & again I have to wait 90 minutes to perform this check.
Is there any way to optimize it so that i can retain the previously refined sources & only update the shell script in the workspace
I am looking to automating in jenkins But being a novice I am stuck how to do it.
Any help would be helpful thanks
Try to download all the stuff to different folder, run 'chown' on the folder to make user 'jenkins' owner of the repo. Set your script to check the folder.
For the test purpose it will work.
Related
I have a job running perfectly in Jenkins called 'MyBuild'. The Source Code is from my Azure Repo and builds everything and even creates Artifacts for me. Perfect up to that point.
The Files that are generated by this 'MyBuild' job come with a default config from the azure repo. So I have a batch script called 'devscript.cmd' on the same PC/server where Jenkins is running. This script makes a few changes to the configs files in the last successful Build. After making those changes it should begin a dotnet run on my application.
So far What I tried is: created another freestyle Job in Jenkins called 'Development_Run'. I left all the configs as default and added the build steps of Execute Windows Batch Commands.
So the Job runs well since it just navigates to that path and Executes the script.
The Problem is. All what the Script is supposed to Do, doesn't happen. NB:// I test the script in the PC first and it runs well before starting the build process in the 'Development_Run' Job. So the Script is executed but in actual sense the contents of the script don't get executed.
Illustration below
Where I could I be going wrong? Or someone help by giving an alternative so as to achieve what I am doing.
The reason for all this is automation. I could do this manually every time but since the development is ongoing I need to have this automated.
We have started to use jenkins from last few months and now the size of home directory is about 50GB. I noticed that size of Jobs and workspace directories are about 20 GB each. How can I clean them? What kind of strategy I should use?
Consider the various Jenkins areas that tend to grow excessively. The key areas are: system logs, job logs, artifact storage and job workspaces. This will detail options to best manage each of these.
System logs
System logs may be found in <JENKINS_HOME>/logs or /var/log/jenkins/jenkins.log, depending on your installation. By default, Jenkins does not always include log rotation (logrotate), especially if running straight from the war. The solution is to add logrotate. This Cloudbees post and my S/O response add details.
You can also set the Jenkins System Property hudson.triggers.SafeTimerTask.logsTargetDir to relocate the logs outside the <JENKINS_HOME>. Why answered later.
Job Logs
Each job has an option to [ X ] Discard old builds. As of LTS 2.222.1, Jenkins introduced a Global Build discarder (pull #4368) with similar options and default actions. This is a global setting, Prior to that, job logs (and artifacts) were retained forever by default (not good).
Advanced options can manage artifact retention (from post-build action, "Archive the artifacts" separately.
What's in Jobs directory?
The Jobs directory contains a directory for every job (and folders if you use them). Inside the directory is the job config.xml (a few KB in size), plus a directory builds. builds has a numbered directory holding the build logs for each retained build, a copy of the config.xml at runtime and possibly some additional files of record (changelog.xml, injectedEnvVars.txt). IF you chose the Archive the artifacts option, there's also an archive directory, which contains the artifacts from that build.
Jenkins System Property, jenkins.model.Jenkins.buildsDir, lets you relocate the builds to outside the <JENKINS_HOME>
Why Relocate logs outside <JENKINS_HOME>?
I would strongly recommend relocating both the system logs and the job / build logs (and artifacts). By moving the system logs and build logs (and artifacts if ticked) outside of <JENKINS_HOME>, what's left is the really important stuff to back and restore Jenkins and jobs in the event of disaster or migration. Carefully read and understand the steps "to support migration of existing build records" to avoid build-related errors. It also makes it much easier to analyze which job logs are consuming all the space and why (ie: logs vs artifacts).
Workspaces
Workspaces are where the source code is checked out and the job (build) is executed. Workspaces should be ephemeral. Best Practicesare to start with an empty workspace and clean up when you are done - use Workspace Cleanup ( cleanWS() ) plugun, unless necessary.
The OP's indication of a workspaces in the Jenkins controller suggests jobs are being run on the master. say that's not a good (or secure) practice, except lightweight pipelines always execute on master. Mis-configured job pipelines will also fall back to master (will try find reference). You can set up a node physically running on the same server as the master for better security.
You can use cleanws() EXCLUDE and INCLUDE patterns to selectively clean the workspace if delete all is not viable.
There are two Jenkins System Properties to control the location of the workspace directory. For the master: jenkins.model.Jenkins.workspacesDir and for the nodes/agents: hudson.model.Slave.workspaceRoot. Again, as these are ephemeral, get them out of <JENKINS_HOME> so you can better manage and monitor.
Finally, one more space consideration...
Both maven and npm cache artifacts in a local repository. Typically that is located in the user's $HOME directory. If incrementing versions often, that content will get stale and bloated. It's a cache, so take a time hit every once in a while and purge it or otherwise mange the content.
However, it's possible to relocate the cache elsewhere through maven and npm settings. Also, if running a maven step, every step has the Advanced option to have a private repository. That is located within in job's workspace. The benefit is you know what your build is using; no contamination. The downside though is massive duplication and wasted space if all jobs have private repos and you never clean them out or delete the workspaces, or longer build times every time if you cleaned. Consider using s the cleanWS() or a separate job to purge as needed.
The workspaces can be cleaned after and/or prior any execution. I recommend doing it prior and after an execution. After the build, do it only on successful builds. In case of errors, you can enter the workspaces and check there for any clue. You do it on the pipeline using the CleanWs() command.
For jobs directory you can select on your jobs the amount of time / maximum of executions to store. This is more complicated because it depends on what you want to save. For example if there is a lot of builds and you don't mind deleting that information you could save 10 builds during 30 days . That configuration is on the job configuration under job properties and search for "Discard old builds" and "Days to keep build" and "Max # builds to keep".
My suggestion is that you use larger numbers at first and then you can test how it behaves
I've been asked to create a CI pipeline for a project at my work, I'm creating a load test with JMeter and Taurus so I plan to integrate it with Jenkins to build all the pipeline. I'm just starting on this field and a question that came to my mind is:
What happens to all the data created by the Load Test? does it goes to the deploy phase or it gets deleted once the test is done, should I clean after the tests end?
The data is being kept in the Jenkins workspace and by default it will be kept in the file system forever.
If you decide to publish the artifacts they will be available at Jenkins build dashboard via the web interface.
You might also be interested in Jenkins Performance Plugin which allows plotting performance trend charts and conditionally marking builds as unstable or failed depending on pass/fail thresholds
Example configuration can be found in the How to Run a Taurus Test through Jenkins Pipelines article
I am not completely familiar with your setup but as far as I can see from a quick research, JMeter does the same as every other testing framework and generates HTML reports. Jenkins wont delete them, unless you explicitly delete them (rm file.html) or call cleanWs (clean workspace). If the job is deleted so are the files.
So the test result file should still be present in the deploy phase. You can use a plugin to collect the result. Or just archive it. Or do whatever fits your workflow.
There is generally no need to clean it up (you usually configure Jenkins to delete old builds which takes care of that)
I have two Jenkins, both are master. Both have 5 salve Jenkins each. I have one job on first jenkins that needs to be cloned for each job.
I can clone the job on first jenkins and its slave but not on second master jenkins. Is there a way to clone a job from one jenkins to another?
I have one more question can I archive the job at some defined location other than master jenkins, May be on slave?
I assume you have a job called "JOB" on "Jenkins1" and you want to copy it to "Jenkins2":
curl JENKINS1_URL/job/JOB/config.xml | java -jar jenkins-cli.war -s JENKINS2_URL create-job
You might need to add username and password if you have turned on security in Jenkins. The jenkins-cli.war is available from your $JENKINS_URL/cli.
Ideally you should make sure you have the same plugins installed on both Jenkins1 and Jenkins2. More similar you can make the two Jenkins masters, the fewer problems you will have importing the the job.
For the second part of your question: slaves don't store any Jenkins configuration. All configuration is done on Master. There is a lot of backup plugins, some backup the whole Jenkins, some backup just job configuration, some backup individual jobs, export them to files, or even store/track changes from SCM such as SVN.
So "archiving job configuration to slave" simply makes no sense. But at the end of the day, a job configuration is simply an .xml file, and you can take that file and copy it anywhere you want.
As for the first part of the question, it's unclear what you want. Do you want to clone a job automatically (as part of another job's process), programmatically (through some script) or manually (through the UI, other means)?
Edit:
Go to your JENKINS_HOME directory on the server filesystem, navigate to the jobs folder, then select the specific job folder that you want.
Copy the config.xml to another server, this will create the same job with the same configuration (make sure your plugins are same)
Copy the whole job_name folder if you want to preserve history, builds, artifacts, etc
I am looking to add some automated tests to run nightly on a project. Currently the project has a few jobs that create multiple builds of various components of the project.
The builds create rpm files, there are multiple jobs creating multiple rpms, I want to grab all of the rpms and install them and test them all under a single test job, there are lots of dependencies on each other. I can install via the command line but these rpms are stored on the Jenkins master machine.
This is as far as I have got;
I have set up the job in Jenkins
I have created a slave for the job to run on
I have used Jenkins to run a bash script on the slave (works)
What I want to do is the following;
At regular intervals (lets say once per day when I know builds have all completed) fetch the most recent passed builds of all the projects and copy them to the slave machine
Install the rpms using a script.
The script performs certain tests during the install (looking at logs etc...) so I want to collect these all and send the results back to Jenkins (may eventually perform other tests here too)
I want the status of the last build image to be determined by my own tests
I also want the test results, logs, etc... to be stored in the Jenkins test job so that we can view them and marvel at their awesomeness!
What I don't know how to do is;
How to copy the files to the slave? Should this be handled on the slave itself using wget or something, or does Jenkins have the functionality (plugin maybe) that handles this all for me?
How do I report my custom results back to the Jenkins job?
I only started working with Jenkins three days ago (the project and Jenkins build jobs are a lot older than that), apologies if I'm missing anything obvious.
UPDATE
I'm thinking a combination of these plugins might do the trick, I've not yet looked into these too much yet though.
Copy artifact plugin to copy the rpms from the latest stable builds of the other jobs
xUnit plugin to interpret some xml files that I generate during the test process
I didn't actually need any plugins for this. I simply set up the job to run on the slave, had a build step that ran some tests and generated an xml file (similar to the jUnit xml results) and then added a post build step to look at the jUnit results (even though the tests weren't jUnit tests).
This worked a charm and I have builds being marked as unstable if they fail tests that I specify, like did they install an rpm and did such and such happen.
I was able to get the latest stable builds as the latest stable builds are coppied to a file server anyway, any failed builds don't go there so that was simple.