We are in the process of switching from VSoft Continua CI to Jenkins for our build management environment. As we use a slightly modified Gitflow process we would like Jenkins to be able to build from any feature, release or hotfix branch and pull requests, we decided to go for the Jenkins Pipeline.
The version number for builds from the release and hotfix branches are based on the branch name (e.g. release/2.1.0) while builds from any other branch or pull request is based on the date (e.g. September 6th 2018 resolves to 18.9.6). Continua CI provides a auto increment build number across all build configurations that is why we use this build number as the final part of our build number (e.g. 2.1.0.10, 18.9.6.11, 2.1.0.12, ...). This generated version number is passed as parameter to MSBuild using this version number as the file version and assembly version of our .NET binaries.
I'm looking for a similar solution in Jenkins. The Jenkins Pipeline assigns a separate auto increment build number per branch and pull request which might lead to two builds from different branches having the same version. I already tried using global environment variables to store the version and increase the value with every build but it seems that global enviroment variables cannot be set from Pipeline tasks.
Is there a way for a Jenkins Pipeline project to share a build number across all branches/pull requests?
Here are a few ideas:
have it file based: have your stage execute on, say, the master node; pick a file and decide on the format (properties file can be a good start); lock, read, update, write, unlock.
delegate this to an external service (for instance, a service with a
REST endpoint that you use to request an ID).
write a plugin for it.
I want to build limited set of Jenkins branches master develop release* hotfix* using Jenkins Github Organization Folder plugin.
The problem is that once I add the filter for the list of branches to build to master develop release* hotfix* Jenkins stops building PRs.
It does work as expected for these branches but I does not pull PRs made against them and this is mandatory as we want to be able to merge only code that was executed by Jenkins, but without having to include the source branches in the normal list of builds (we don't want to build each feature branch as they can get lots and lots of changes.)
After two weeks and lots and lots of tests I finally raised a bug
https://issues.jenkins-ci.org/browse/JENKINS-36352
This seems to be a bug so far.
I am looking for implementation of CI/CD in to my current project here is what i think will work.
Environment consists of
- Jenkins
- git
- docker
- gradle
- Linux servers
- Sonar
- Ansible.
Each tool will be used as following.
Git:- Developers will push there code to this CVS.
Jenkins:- On detecting Check-in Jenkins will trigger a build and will deploy to one of the server.
Sonar:- will be used for code coverage and will check the code before building the same through Jenkins.
ansible:- ansible will be used to quickly prepare added nodes so that code can be deployed to them.
Docker in case if we need fresh test environments every time we can use docker+ ansible combo for doing the things.
Flow of work will be
User run unit test cases on his machine and commits the code to the server.
Jenkins will pull the code from git and will run sonar on the same and will generate reports.
jenkins will create build and will deploy the same on dev server.
A jenkins job will run and will perform the integration testing on the dev server
Any other automated tests can be run.
Finally builds pushed to next server using Jenkins.
I will use shell commands inside Jenkins to push compiled code from one server to another.
In my this scenario can some one answer me following.
Where will sonar get fit and how to use the same?
I see there are CD tools, cant i push compiled code to the servers using shell scripts written inside the Jenkins jobs to automatically deploy the things? What extra benefits a CD tool provides
Is is wise to create fresh test environment or we can keep using the old one again and again?
Will this complete CI/CD?
can someone share there implementation
You say you plan to use Git. I'll outline a scenario using Git on GitHub
Developers push code changes here as pull requests
The SonarQube GitHub Plugin kicks off an initial analysis of only the code changed in the PR looking for the introduction of new issues (note that coverage and duplications are not included in this check)
Once the PR is merged, Jenkins (in one job or several, depending on your needs)
builds
fires integration tests & any other automated tests
runs SonarQube Scan. Note that this comes last to include integration test results.
pushes build to next server
Note that the ability to break the build when the project doesn't pass the SonarQube Quality Gate you've set up may be desirable in your situation. Unfortunately, it's not available in the current server version: 5.2. It is available in 5.1, and is should return soon.
We work on .NET project, using TFS for:
source control
builds: gated check-ins that produce MSI files
deployments to Labs
We want to create a proper Continuous Delivery pipeline, that is a Dashboard with pipelines for each check-in with traffic lights.
Pipeline should show all the stages like TFS build > Deploy to Lab > Smoke test > Integration Tests > Acceptance Tests > Deploy to PreProd > ...
So it has to be tightly integrated with TFS.
We are assessing 2 options:
use TFS-based tool\plugin\dashboard if there are any that can show pipelines?
use CI tool for example Jenkins, TeamCity, Bamboo to build this pipeline - ideally with support to fetch built code from TFS drop folder, not just the source code
What would you recommend?
If you are using TFS why don't you leverage the built in Release Management tooling? You can create a release pipeline that is automted and even include approvals I necessary.
http://nakedalm.com/building-release-pipeline-release-management-visual-studio-2013/
If you want to integrate the lab tools for collecting test results as part of your pipeline this works as well.
http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
This works pretty well and the new features anounced at Connect() will make it even better.
Use case:
CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.
Which CI server is able to handle this?
My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.
I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)
The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges
What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM:
https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement
This question answers how to make Jenkins build on every subversion commit:
Jenkins CI: How to trigger builds on SVN commit
TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/
It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.
It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers
By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.
I dont want that CI server would launch jobs in parallel to avoid
performance issues and crashes in cases of many simultanious jobs.
In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks
As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:
Modeling Parallel Processes: Steps
svn up
configure
make
make test
make dist
In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.
From the docs:
The Triggerable scheduler waits to be triggered by a Trigger step (see
Triggering Schedulers) in another build. That step can optionally wait
for the scheduler's builds to complete. This provides two advantages
over Dependent schedulers.
References:
how to lock steps in buildbot
Coordinating Parallel Builds with
Buildbot
There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:
It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)
There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.
www.gitlab.com
www.gitlab.com/gitlab-ci
To only run tests once for every revision you can do something like this:
build
post-build
check if the revision of the build is in /tmp/jenkins-test-run
if the revision is in the file skip tests
if the revision is NOT in the file run tests
if we ran the tests then write the ID in /tmp/jenkins-test-run