I'm currently using Jenkins to trigger tests via an external test system and aggregate results. This system follows a typical master slave relationship and stores results, logs, and other files generated from these tests on the slave machine.
Is there a way to push these files from the test slave to the Jenkins workspace while the job is running? This way they can be available to the developer with the test results on job completion.
This would be a pull rather than a push. You would have Jenkins job pull the files in any method you prefer (scp, rsync, wget etc).
Jenkins workspace locations are typically randomised folders based on the job name with a hash/incrementor number in the path, so a Push would be inadvisable.
Related
Who does the actual cloning of the project, is it the master or the agent node? If it is the master, then how does the agent node actually execute the job. If it is the agent node, how can we view the workspace in the browser?
When people ask "where is the workspace" the answer is usually a path, but I am more interested in where that path is, on the master or the agent node? Or maybe it is both?
Edit1
Aligned terminology to this: https://jenkins.io/doc/book/glossary/ in order to avoid confusion.
In a Jenkins set up all the machines are considered nodes. The master node connects to one or more agent nodes. Executors can run both on the master or agent nodes.
In my scenario, no executors run on the master. They are run only on the agent nodes.
The answer is: it depends !
First of all, although it is not a good practice IMO, some installation let the master be an actual worker and run jobs. In this case, the workspace will be on the master.
If you configured the master not to accept jobs, there are still occasion when a workspace can be created on the master. A good example is when your job is a "pipeline script from SCM". In this case, the master will create a workspace for the job, clone the target repo, read the pipeline, and start needed jobs on whatever slave is targeted, creating a workspace to run the actions themselves. If the pipeline targets multiple slaves, there will be a workspace on each of them.
In simple situation (e.g. maven or freestyle job), the workspace will only be on the targeted slave.
I needed to dig a bit deeper to understand this.
I ran a brand new instance of Jenkins and I attached a single agent node. I used SSH and I set the remote (agent) root directory to: /home/igorski/jenkins
As soon as I attached the node the remoting folder and remoting.jar showed up in that root directory.
I ran a basic Gradle Java pipeline job (Jenkinsfile in the project).
The workspace showed up on the slave. Not on the master.
From the Jenkins GUI I can access the workspace and see it's contents.
At the moment I kill the agent machine I can no longer view the workspace in Jenkins.
My guess is that the remoting.jar somehow does a live sync.
I also ran a freestyle project and I can confirm the same. As soon as the agent is killed I can no longer open the Workspace and I get an error stack trace:
hudson.remoting.Channel$CallSiteStackTrace: Remote call to JenkoOne
This was much more obvious with the Pipeline job though. There you get a link to the agent that you need to click in order to see the contents. As soon as the agent is gone the link is disabled. And you know exactly on which agent the node is. With freestyle jobs, you just get a Workspace link. There is no indication on what agent it is or if the agent is accessible at the moment.
So, both Zeitounator and fabian were correct.
I have a bunch of pipeline jobs, yet when executed, workspaces of some get preserved, some are deleted. How does jenkins make these decisions?
Based on my findings so far:
All jobs executed on nodes will have their workspace persisted, e.g. /home/ec2-user/workspaces/some-job
Some works on master keep their workspaces but some others' workspaces disappear after the job has finished. For example, after my build job succeeded, if I ssh in I can see the its workspace directory; but all my e2e jobs have no workspace.
Note I didn't use any of clearWs, deleteDir etc in my pipelines.
By the way, the reason I'm looking into workspaces is the disk usage keeps increasing and I want to cleanup. I thought the workspace is overwritten each time a job runs, but yet I get the 'Disk space is too low' warning several times.
Jenkins is creating a new workspace for every build job (= run) per default. You can see that in the path of the ws in your console log: /here/is/the/ws#buildnumber. If you dont want to have that behavior you can set it to an path which is for instance for every repo the same: How to set specific workspace folder for jenkins multibranch pipeline projects
Maybe some of your jobs don't get executed on the Jenkins Master, but on some connected Node (via an agent directive within your Jenkinsfile or Pipeline description). If that's the case you won't see a build directory inside the workspace for this Job on the Jenkins master, but on the connected Node.
You would only get the build results (like artifacts, reports, etc.) under /<JENKINS_HOME>/jobs/My_Job/ on the Master.
Remember that you could trigger a Jenkins build on a node also indirectly if you, for example, run the build within a Dockerfile and have configured (within Jenkins configuration) a specific node label for execution of Docker builds.
I would like to set up jenkins server that would run test scripts based on successful build deployments on other Jenkins servers. for example, if the QA jenkins server is named JQA1OnMachine1 and i have three others that are named
J2OnMachine2, J3OnMachine3, J4OnMachine4 (different jenkins server on different boxes) can the JQA1OnMachine1 (QA jenkis) poll the others at regular interval to see if a build was deployed successfully? if so can anyone tell me how?
Jenkins master slave along with Jenkins Pipeline Plugin would be one of the better ways to implement this however, since you don't want to use that approach you can explore PSTools to remotely capture processes or files on different server.
Your builds may update a file on the build server post completion of the build and your QA machine can run script with PSTools to monitor and trigger the QA testing based on the file content
I'm trying to a run a simple windows batch command (say copy) on Master inside a job that's set to run on a particular slave.
What i'm trying to accomplish with this is that copy the build log that gets saved on a master to a shared drive that's accessible from master. Please advise.
You are going to have to make the jenkins file system visible on the client independently of Jenkins. Since you have a Windows client, you are probably going to have to setup sharing from the Jenkins master using samba or something of the sort.
What I do instead: When I need assets from master is I use curl or wget to download the assets to the clients. You can use the FSTrigger plugin to start builds when the file changes on the Jenkins master. Once curl or wget has run, your asset is then in the %WORKSPACE% directory and you can proceed.
I would recommend to handle the logfile copying (and maybe further tasks) as a dedicated job (let's call it "SaveLog"). SaveLog should be tied to run only on master.
You should then configure SaveLog to be triggered after completion of your primary job.
The Logfile is already available on master, even if you do not save any artifacts.
Should you need further files from the slave workspace, then you should save them as artifacts. SaveLog (on master) can then still decide whether to do anything useful with those artifacts.
I have two Jenkins, both are master. Both have 5 salve Jenkins each. I have one job on first jenkins that needs to be cloned for each job.
I can clone the job on first jenkins and its slave but not on second master jenkins. Is there a way to clone a job from one jenkins to another?
I have one more question can I archive the job at some defined location other than master jenkins, May be on slave?
I assume you have a job called "JOB" on "Jenkins1" and you want to copy it to "Jenkins2":
curl JENKINS1_URL/job/JOB/config.xml | java -jar jenkins-cli.war -s JENKINS2_URL create-job
You might need to add username and password if you have turned on security in Jenkins. The jenkins-cli.war is available from your $JENKINS_URL/cli.
Ideally you should make sure you have the same plugins installed on both Jenkins1 and Jenkins2. More similar you can make the two Jenkins masters, the fewer problems you will have importing the the job.
For the second part of your question: slaves don't store any Jenkins configuration. All configuration is done on Master. There is a lot of backup plugins, some backup the whole Jenkins, some backup just job configuration, some backup individual jobs, export them to files, or even store/track changes from SCM such as SVN.
So "archiving job configuration to slave" simply makes no sense. But at the end of the day, a job configuration is simply an .xml file, and you can take that file and copy it anywhere you want.
As for the first part of the question, it's unclear what you want. Do you want to clone a job automatically (as part of another job's process), programmatically (through some script) or manually (through the UI, other means)?
Edit:
Go to your JENKINS_HOME directory on the server filesystem, navigate to the jobs folder, then select the specific job folder that you want.
Copy the config.xml to another server, this will create the same job with the same configuration (make sure your plugins are same)
Copy the whole job_name folder if you want to preserve history, builds, artifacts, etc