I have a project where part of the build process is to create a native library on a remote machine. This is currently a manual process outside of the CI builds made by Jenkins.
The setup in question is that the Jenkins master server build a GIT based maven project, which has a dependency to a native library which can only be built on a specific machine. Jenkins can't compile this module, and because of this, it is currently a manual process.
I would like to install a Jenkins slave on the machine that creates the native library, and returns the compiled files to the Jenkins master, without handling any other parts of the build.
I am having trouble figuring out if this is even possible. The number of articles i have found on the subject discusses Jenkins slaves as a means of distributing the build, but i want the slave to take responsibility for a small part of the build process, and nothing else. The Jenkins master should just send the build request to the slave and wait for the result, instead of trying to compile the code itself.
I do exactly the same. My setup, very similar to what Mark O'Connor and gaige are advising, and I am using the Copy Artifact plugin.
job A: produces a zip file on a Mac
job B, runs on slave B - Windows machine, takes the zip as input and produces an MSI
Here's the important part in the config of job B:
restrict the job B on the proper slave using labels
make sure job B happens after job A
make sure artifacts from job A are sent to job B before your build
build your stuff
archive artifacts produced by job B
Delegating part of a job to a slave is something that would have to be done external to Jenkins, for example, using ssh.
However, as #kan indicates, you most likely want to extract the native library build as a separate job and then have that job execute on a particular slave, or any slave that meets a specific criteria.
To do this, my suggestion would be to use Labels in the node configurations to determine which slaves can be used for building that particular job.
In Jenkins > nodes > <slave node>, use the Labels property to set one-word labels that indicate your specific requirements, such as the OS or processor type.
Then, in the jobs that are node-specific, check Restrict where this project can be run and set the Label Expression to something that meets your criteria. If the criteria is simple, it will just be a single word, if you need a boolean, you can use those as well (such as OSX&&Lion in our case).
I believe this is all in the standard version of Jenkins, without need for a special plugin. Leave me a comment if it isn't and I'll try and diagnose which plugin enables this functionality.
This is problem is solved by using a binary repository manager to centralize your software artifacts. Personally I use Nexus, but it could be something as dumb as a remote file system.
The idea is to publish the built artifact after each Jenkins job (if you don't like Nexus, you could use one of the Publish over plugins) and retrieve it as a build dependency in the next job.
This approach means it longer matters where the build executes, and has the added advantage of decoupling the build of each module component.
Related
Why is Jenkins changing node to do another build when it doesn't need to?
We have a Jenkins setup with 3 Mac and 3 Windows nodes, building free style projects. We do not use the master for builds. The projects are set to run on any of the 3 nodes suitable for their platform. We are using labels for this purpose.
Some of the time, when we do a build it will do the build on the same node as last time.
But sometimes, without any obvious pattern, it will change to a different node, even though the previously used node is available and not busy. This wastes a lot of time potentially as incremental builds on the same node are much faster than pulling and building everything from scratch.
Jenkins claims it should allocate jobs to the same node if possible, when multiple possibilities exist.
As a result, from the user’s point of view, it looks as if Jenkins tries to always use the same node for the same job, unless it’s not available, in which case it’ll build elsewhere. But as soon as the preferred node is available, the build comes back to it. Reference
Jenkins version is 2.289.2 presently.
The jobs are freestyle builds with shell script/command prompt steps.
Repositories are Git and Mercurial.
I think if you read the Restrict where this project can be run that might provide you answer to your question.
As its definition tells By default, builds of this project may be executed on any agents that are available and configured to accept new builds.
So if you don't specify the agent and restrict the build to be run a particular node, it will randomly choose.
This wastes a lot of time potentially as incremental builds on the
same node are much faster than pulling and building everything from
scratch.
To solve the above problem only, we have the Restrict where this project can be run feature in Jenkins. Please look to the below screenshot for detailed help that we can get in Jenkins .
Following the Jenkins Best Practices, I want to avoid that Build Jobs/Pipelines could be executed into my Jenkins Master.
To do so, I've installed the Job Restrictions Plugin, using it to configure the Master to run only some Maintenance Pipelines.
The problem is that now Build Pipelines that are configured to run on specific Agents, are not executed anymore. I see that the Build Queue continuously grows, and the Pipelines are not runned. I think that this behaviour could be related to Flyweight Executors of the Master.
So, the question is the following: How can I execute on Master just a little subset of Maintenance Pipelines and, in the mean time, execute Build Pipelines only on specific Agent?
You can configure the master node to only be used when explicitly named. Just click the master node > go to configure and change Use this node as much as possible to Only build jobs with label expressions matching this node
I found the solution that perfectly fits with my needs, here.
To quickly sum up the solution, I was to able to exclude all the user Builds from Master and run on it only the Jobs/Pipelines of a specific Jenkins folder (IuA in my case), configuring the Job Restrictions Plugin in the following way:
In order to better understand the logic behind this solution, I recommend you to give a look at link that I posted above.
I want to have one Jenkins job control the build number of another job but without the inconvenience of reloading the entire project configuration from disk. I have seen that it's easily possible to directly update the nextBuildNumber file of the target job (I can do this as a build step of Job A) but this does not take effect immediately. Restarting Jenkins or even reloading the Jenkins configs from disk takes way too long and can only be done when there are no builds in progress.
I have tried the groovy script mentioned in the below post by running it from the Manage Jenkins > Script Console. The same post also suggests the script can be saved as a file and run from the CLI. Can it be run from a build step?
I want Job A to determine Job B's next build number and set it so that Job B can run (later in the same day) with the desired build number.
https://stackoverflow.com/a/20077362/4306857
Perhaps I should clarify. I'm not familiar with Groovy so I'm looking at the various build step options like "Execute Windows batch command" which I have a lot of experience with. I can see an "Invoke Gradle script" option so I was wondering if there may be a plugin that can run Groovy scripts perhaps?
The reason this requirement has arisen is that we are compiling a product for two different platforms. We want to compile the codebase almost simultaneously for both platforms with two jobs (A & B) which will both update the JIRA cases included in the builds. We feel it will be less confusing to have both these jobs running with the same build number so that when we talk about a particular issue being addressed in build #75, say, we won't have to qualify that by stating the full job name. If JOB-A build #75 and JOB-B build #75 are both compiled on the same day from the same codebase we can test and compare the results of both builds with far less confusion than if the build numbers go out of sync.
Obviously, in the short term we will use the Set Next Build Number plugin to manually keep the build numbers in step but we want to automate this if possible.
Depends on whether or not you are using Version Number plugin:
[ X ] Create a formatted version number
Build Display Name [ X ] Use the formatted version number for build display name.
Assuming you are NOT, this groovy script will do:
def NextNumber=42
job=Jenkins.instance.getItemByFullName('path/to/jobName')
job.nextBuildNumber = NextNumber
job.save();
You will need groovy plugin for that. Place that in an "Execute system Groovy script" step. Make sure to choose system groovy. That will execute on the master, where the job config and metadata is stored so you have access to the Jenkins internals and data.
I'd suggest you should really be using the above options rather than relying on "keeping both jobs in sync" via a script or manually. You can then pass the label to be used from the first job as a parameter to the second job. That would also require Parameterized Trigger as well as Version Number plugins.
You can even use ${BUILD_DATE_FORMATTED} or ${BUILD_TIMESTAMP}, etc.
Postdate: thinking about the problemspace from a different perspective, that of running 2+ builds on different platforms (simultaneously), there's a plugin for that: Matrix project. You can run it as a freeatyle job on multiple nodes or is excellently described as Matrix building in scripted pipeline. Not sure how that would tie in to JIRA.
There is a job controlled by Development team which built in a different node. I am on Testing team who want to take the artifacts and deploy on test device.
I can see those Artifacts from dev are stored in some path in dev's node. Does it means it must first archived in Jenkins master before I can copy it to my job?
I am using Copy Artifact plugin and constantly getting the error
Failed to copy artifacts from <dev-job> with filter: <path-in-dev-node>
*Some newbie question since i just moved from TeamCity
You probably want to use: Copy Artifact plugin.
Adds a build step to copy artifacts from another project.
Consider also, the Jenkins post-buid step "Archive the artifacts".
If you copy from the other job's workspace, what happens if another job is in progress or the workspace is wiped? That step copies them from the node to the master and stores a copy along with the build logs, etc. That makes them available via the UI as long as the build logs remain. It can take up space tho.
If you do use archive artifacts, consider using the system property jenkins.model.Jenkins.buildsDir to store all the build logs (and artifacts) outside of the jobs config directory. Some downtime and work required to separate the two (config / logs) .
You may also want to consider using a proper repository manager (Nexus / artifactory)
Finally, you may want to learn about using a Jenkins pipeline rather the relying on chained jobs, triggers or users and so forth. Why? 'cos it's much more controlled and easier to maintain.
ps: I'm not a huge fan of artifactDeployer, but it may work for you.
pps: you may want to review this in depth answer: Jenkis downstream job fails to find upstream artifacts
I am in a process of configuring Jenkins to deploy artifacts. I only need apache ant and java to create artifacts(both are available on the host machine) and no other external libraries. So, I think using Maven will make it unnecessarily complex as I have only 2 ant files. I want to keep it as simple as possible.
What I want to achieve is:
1. Trigger a Jenkins job 'A' to build the artifact and deploy it to nexus repository.
2. Trigger another Jenkins Job 'B' to take same artifact generated in in step above and deploy it to target environment.
Can anyone please help me to identify challenges with my approach and share some useful links to achieve what I have specified.
A short answer is Yes you can. Each of the component you mentioned can be used individually and can be integrated into your build pipeline. TBH, your use case isn't one off and can be easily done if you start here.