When using Bazel in a multi repo environment what's the best strategy for maintaining dependency consistency?
Eg. Workspaces (individual git repos) A, B, C all depend on D. When the version of D changes, I want A, B and C to all be on the same version of D with the minimal amount of work.
You can use git_repository rule in A,B, and C's WORKSPACE file:
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "D",
remote = "https://github.com/my_org/d",
branch = "master",
)
Notice that the branch attribute is set to master and not some specific commit. The branch doesn't have to be master. It could be any branch.
If you want a way to reproduce builds,
this can be done by using bazel sync command.
There is a bazel blog post with detailed explanation of the steps you need to take to have a "snapshot" file of your external workspaces (resolved.bzl).
You can use it either as another build artifact for future investigations or to have more control over when to update the "pointers" to the external workspace by calling bazel sync with --experimental_repository_resolved_file=resolved.bzl
Related
I have uploaded some release artifacts from Jenkins to Nexus. I can see they have been created under Nexus as per the -Dversion I have provided.
Example: AppName-BuildNumber-Snapshot.zip
However, somehow the repository path is autogenerated, it has extra yyyymmdd.hhmmss appended to it.
Example: AppName-BuildNumber-Snapshot-yyyymmdd.hhmmss.zip
I would like to download the release artifact from Jenkins but due to the autogenerated Nexus repository path I am not able to download it in the separate job.
How can I force it to stop adding yyyymmdd.hhmmss to the repository path? Or is there anyway I can retrieve the repository path using AppName and BuildNumber?
If you are uploading a SNAPSHOT, then Nexus dynamically stores it with a timestamp. That's because you can replace -SNAPSHOTs with newer copy w/same GAV. This is unlike real # jars which cannot/should not be redeployed, thus preserving their integrity.
To download a -SNAPSHOT, you just reference it as -SNAPSHOT; maven, aether, etc will retrieve the latest.
Maven (by default) only checks the remote repository for versions newer than your local once daily, unless you use -U option or change the updatePolicy.
If you want "solid" number, then you must release jar.
btw, you should see it's -yyyymmdd.hhmmss-nn, where nn is an incremental number for that version.
Nexus supports many different repository formats. If you only require maven, use Nexus 2 as it better supports maven.
Repository types: Maven has two distinct types:Release and Snapshot Repositories.
If you want to "upload a zip file containing release artifacts and retrieve it back based on the build version number?", then that's what you should do.
You must configure a repository of type release, not snapshots (gleened from: -DrepositoryId=dsnexus-snapshots) (or in addition to type snapshots).
To pass the "build version number", then presumably, you have:
[ X ] Create a formatted version number
Environment Variable Name [ label ]
Make sure your maven step has:
[ X ] Inject build variables
Then, mvn deploy:deploy-file -DgroupId=com.my.gid -DartifactId=AppName -Dversion=${label} -DrepositoryId=dsnexus-release
ps: you'll also want to implement a cleanup strategy in your Nexus repository if you are pumping every build as a new artifact.
I need to build two Maven projects which are available on Github. The first project produces a library which is needed by the second.
What is the recommended option to do it in Jenkins ?
Create a shell script which checkout and build both projects (in the correct order).
Use a pipeline ?
If you could provide any example/ relevant link that would be great. Thanks
You can achieve using 2 (upto my knowledge) options. Lets consider project A which builds libraries and project B is the dependent. I described below two options by considering two constraints.
Project B should be built whenever Project A is built
Project B can also be built alone if there is only changes with Project B but not with A
Option 1:
You have to create two "Maven Build jobs" for Project A & B. In Project B, you have to specify "Build whenever a SNAPSHOT dependency is built" under "Build Triggers".
This only applies when project B pom.xml has dependency of Project A built artifacts and the artifacts are SNAPSHOT. Below is the image of the same.
Options 2: You can create two Freestyle job for both Project A & B. Define Project B is downstream for Project A ( It can also be done vice versa ) using "Build Other Projects" under "Post-Build Options" of Project A.
And you can copy artifacts from Project A to Project B using Copy Artifact Plugin
Exception: Ideally the above step is not necessary, if your Project A artifacts are installed on $HOME/.m2/repository and Project B dependency defined in pom.xml (or) both project A & B are using Private Maven Repository option.
Assumption: I hope using single pipeline, it will be hard to manage this scenario. Though I expect somebody else will write with pipeline example to enlighten me as well :)
You can achieve this using Jenkins pipeline jobs.
First we need to create a folder for project A and in side that you can download the source code using GIT plugin and you can build the solution.
And create another folder and inside your download the source code of another project and you can build the second solution.
eg:
node{
stage('first project'){
dir('project1'){
git branch: '<Branch>', changelog: false, poll: false, url: '<First Repo URL>'
sh 'mvn clean install'
}
}
stage('second project'){
dir('project2'){
git branch: '<Branch>', changelog: false, poll: false, url: '<Second Repo URL>'
sh 'mvn clean install'
}
}
}
Sharing a method I used for building two changes.
Just add "Depends: xxxxxxx" before the Change-Id in your git commit message:
Depends: fe1a5effxxxxxxxxxxxxxxxxb8xxxx
Change-Id: Ig43j5cxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I'm trying to understand the purpose of the archive directory.
From what I've seen, the archive is per build. Meaning if I set the 'Max # of builds to keep' to 3 for example, The archive will also be deleted.
Is that correct?
If so, how can I tell Jenkins to only save my artifact (zip file) created and delete the build directory?
Is there a one archive directory for all builds?
There are two directories for a Jenkins job, by default (there can be just one, builds, in case of a Workflow job, or more, e.g. an additional promotions directory):
JENKINS_HOME/jobs/${JOB_NAME}/builds/ containing sub-dirs according to $BUILD_ID
JENKINS_HOME/jobs/${JOB_NAME}/workspace/
So, there's one workspace per job and one build directory per build.
Don't let the page Administering Jenkins with this change driven by JENKINS-8446 confuse you. The workspace default, as also mentioned in the inline help (of the current v1.635 at the time of this writing), is still:
Manage Jenkins → Configure System → Advanced... → Workspace Root Directory: ${ITEM_ROOTDIR}/workspace
It's NOT ${JENKINS_HOME}/workspace/${JOB_NAME}.
If a discard of old builds takes place the workspace isn't discarded, but, since it exists per job, it is overwritten with every new build. Hence, if you'd like to keep old artifacts you can:
create an "archive" job that uses the Copy Artifact Plugin and connect it as downstream to your build job.
move them to some other location yourself (including creation of uniquely-named sub-dirs to store the artifact(s) in) using a Post-build Action → Groovy Postbuild or → Execute a set of scripts.
There's also the Discard Old Build plugin which enhances Jenkins' built-in discard functionality via a Post-build Action.
I have set my workspace directory to C:\jenkins_builds\workspace and I want to build ProjA and ProjB, each having a local workfolder (same as project name).
When fetching the source code from my repository, the first two things the TFS plugin does are:
tf workspace -new %workspace-name-A%;%user-name% -server:%my-server%
tf workfold -map $%branch% ProjA -workspace:%workspace-name-A% -server:%my-server%
Which goes fine when building ProjA. The problem is, the first command maps the root directory from the repository directly to my C:\jenkins_builds\workspace directory. The second command does what I actually want, i.e. mapping %branch% to the ProjA subfolder. Later on, when building ProjB, the first command fails (and consequently the build) with the following error message:
The path C:\jenkins_builds\workspace is already mapped in workspace %workspace-name-A%;%user-name%.
OK, it seems like a bad idea to map the root directory to the work directory. But why does this automatically happen when the TFS plugin runs the workspace new line? Currently I have to clean things up between building ProjA and ProjB by running the -unmap command.
My team is using Team Foundation 3.0.
We have the same situation and there are 2 ways to solve this:
use different workspace-root-directories for the two builds
This results in the need for two checkouts => double the space and slower, but better isolation between the two builds
"hardcode" the workspace name to the same for both builds
By default jenkins creates a workspace containing the build name, which can be changed in the "advanced" section of the TFS config, and then you can use the same workspace-/workfolder-mapping for several builds - in our case we called them ProjectName_${NODE_NAME} so it even works on several nodes
We have a maven-3 projects with dependencies like this A.jar <- B.jar <- C.jar <- D.war (<- means depends on). They all are under seperate SVN locations. So my question is , How can I organize Jenkins build job for D.war to force also build fresh jars for A,B and C? Also sometimes I need to build only B.jar and force to build A fresh jar. Maven simple <dependency> tag is not enough to force build them.
Thanks,
Arsen