Does anyone have specific experiences with using TFS 2008/2010 AND Jenkins for Continuous Integration (CI)? We are trying to decide which CI server to use. Our team works exclusively in Microsoft .NET/Visual Studio 2010/C#. We have the following requirements:
Automatically build our web project on every checkin.
Run unit tests with each build.
Automatically deploy green builds to development and/or test environments.
Provide pretty reports.
Provide build/deployment notifications via email.
I realize that installing a tool won't necessarily give us this functionality out-of-the-box and that we will have to integrate with other tools like MSBuild to achieve this.
I'm looking for specific features that Jenkins has that TFS 2008/2010 does not or vice versa. Also which is easier to maintain, use, etc.
I would highly recommend using Jenkins - it will do all of your requirements out of the box apart from possibly #3, but if you can script your deployments then it can do that as well.
Here are some links to help you get your builds up and running:
Blog about doing .NET builds in Jenkins
Jenkins Windows installers
Installing the Jenkins master and slaves as Windows services
Disclaimer: I have no experience with TFS, but I think open solutions are nearly always more flexible and extensible (and cheaper !) than proprietary products.
Late to this game, but I have used both TFS 2010 and Jenkins for CI. TFS 2010 has minimum set of CI tools in it. However, when you want to create a CI pipeline, it's a completely different story while Jenkins can easily create the pipeline.
If you are looking at only CI for one build either one should work. However, when it comes to the entire pipeline, Jenkins is way to go. With TFS it can be done, but Jenkins is better choice.
Here's quick bullet points:
TFS:
With a build definition you can compile, execute tests, return changeset/workitems, send an email when a build is broken
natural integration with visual studio
extremely hard to create CI pipeline. Requires custom handler and extensive workflow work. Not as intuitive as creating a build definition.
Because of the 3rd bullet, it's not easy to maintain/customize/scale CI pipeline
Jenkins:
Need to create a msbuild config file for CI, which is not much pain comparing to creating CI pipeline using TFS. However, TFS gives better/easier tool to create a build definition. however, it is not bad creating config file for msbuild for a project.
Creating a CI pipeline is very easy. Just chain them using upstream/downstream jenkins job trigger and passing an artifact from previous job.
Since Jenkins is very flexible, it is easy to create a jenkins plugin to meet your own needs and provide it to opensource community :)
In summary, if you need complete automated build, test, and deployment system go with Jenkins. If you just need only build and test, TFS might give you an edge over Jenkins.
If you're using Team 2010-2012, there is no reason whatsoever to bring in Jenkins. Team has all the features you listed, and the build process is ridiculously flexible.
Note that if you are stuck on Team 2008 or earlier, you should seriously look at Jenkins -- 2008 and earlier are quite primitive and inflexible compared to 2010 or later.
Related
CICD Process with Jenkins, Bitbucket, SSDT(SQL Server Data Tools).
Please list out the steps to perform CICD Process.
Including what plugins i need to install in Jenkins for SSDT(SSIS-ISPAC file) or SQL Database Solution(DAPAC file)
This question is very broad and as with all the stuff related to databases the best answer would be "it depends". As far as I know that there is no proper plugins either for Jenkins nor for Bitbucket that work with SSDT very well so you'll need to implement all your actions by yourself.
It will depend on your system how the pipeline should look like. There are a lot of questions that you'll need to answer first but without knowing your exact situation it is very hard to suggest you something specific. Example questions:
How many environments do you have?
Do you have tests?
Can somebody change the state of the destination database manually, passing the CI/CD pipeline?
Will you run publish on every commit?
Do you trust what SSDT will decide how to publish database? (Mostly people would like to preview the script that would be executed on prod)
Then after answering these questions you'll might know what will you need. After that you need prepare the proper publish script, exclude/ignore/add object types you'd like to deploy and use MSBuild.exe and SQLPackage.exe command line utilities. You'll run these utilities with specific set of arguments and paths to the publish configs, DACPACs, etc. Bamboo and Jenkins supports command line commands for that.
Is MSBuild being deprecated in favor of TFS/Azure DevOps Build pipelines? I have some custom build steps I need to add, such as obfuscating code and computing file hashes. I'm not sure if I should add them as a custom target in my MSBuild/csproj file or if they should be separate steps in a build pipeline. Has Microsoft provided any guidance about which one to use and/or are there common best practices used by the .NET community?
What are the pros/cons of doing custom build steps in MSBuild vs as a build pipeline step?
No, it's not deprecated. Build pipelines are just glorified task runners. All those pipelines do is run your MSBuild script; they don't actually "build" anything in the same sense that MSBuild builds things.
That said, here are some scenarios where it makes sense to add additional build targets and conditions to an MSBuild file:
It needs to happen during local development
It needs to happen across multiple CI solutions (i.e. Jenkins, TeamCity, Azure DevOps)
You foresee switching to a different CI solution in the future
That should make the pros of MSBuild pretty obvious. It's going to work anywhere you build your application, including on your local desktop, and it's going to be portable. It also has the advantage of being automatically tracked by version control, which (unless you're using YAML build), Azure DevOps builds are not.
In my project we are using QC to execute our test cases(QTP), moving forward we would be eliminating QC (for cost reasons).
As far as I explored MSBuild & Jenkins, they would be suitable.
But MSBuild will trigger the execution when a new build pushed to the repository. Also it will automatically test on the latest build.
Is there any other CI tool available to execute test cases through QTP?
I will be executing automation once in a release. Also we install our application by manual since it requires lots of configuration.
Take a look at HP Application Automation Tools.
This plugin basically replaces the need for QC, and is developed by HP.
Create a Jenkins job using this plugin on the same Jenkins installation used to build your code, then you can configure your job to run your tests as soon the code is available (e.g. on a nightly basis).
See here for a helpful guide on how to implement a simple Jenkins job using this plugin.
They also host the code on Github, which is very useful if you need to change the behavior of the plugin to suit your needs.
Right now we have Jenkins up and running on a single VM. It fetches the code from Github, builds and runs all Unit- and E2E Tests.
This takes 3+ hours.
The thing is, we have some integration tests that restore a test database. There are a lot of tests so it takes a long time.
We want to speed this up significantly.
So I created a Template Slave VM on Azure. It has Visual Studio, IIS, SQL Express, Git and everything else we would need for a deployment.
I can now clone this template to 5-6 identical slaves. Each will be instructed by the Jenkins master to build, deploy and test a chunk (suites) of the codebase.
Is this really the best infrastructure for this? It's a LOT of work setting this up. I have trouble finding good reading material on this subject.
I think you are referring to something called not only Continuous Integration, but more Release Management. Microsoft has the development platform Visual Studio Team Services that can be used in integration with Jenkins mode or/and you can create the build definition and release management definition and automate that.
Sure, it will involve some work (maybe a lot) but using VSTS + Azure (as they are tightly integrated) you can automate it for a future use.
I'm looking to get TFS to build and deploy to our manual and auto TEST environments.
manual TEST - overnight updates
auto TEST - continuous integration
Once deployed, I want TFS to deploy and run a set of web-integration tests (SpecFlow / Selenium / nUnit) on another server (acting pseudo client of the website).
I then want TFS to collate the results of these tests and report back.
What's the best approach to get this working?
You'd normally change the build process template to do what you want. Adding steps to deploy could be one of these customozations. Have a look at the Build Release Deploy build template from the Microsoft ALM Rangers which includes steps to publish the build to another machine.
You can also make use of WebDeploy and add a set of publish profiles to your solution. Team Build will happily execute those steps during build, if told to do so.
In TFS you can use a Test Agent to then execute tests from a remote machine and have the results added to the test results for your build.
Seeing that you'd like to use NUnit ad SpecFlow I suggest using TFS 2012 in combination with Visual Studio 2012, which have built-in support for executing 3rd party based tests. You'll need to install the NUnit adapter and put the assemblies for it in source control so that Team Build will pick them up.