I have set up a Jenkins project using the Perforce Plugin and am now working on configuring the "Publish JUnit test result report" post-build action so that it will run build verification tests upon successful builds.
It's telling me that I need to "first set up your build to run tests" - but I am unsure if I should be doing this via Visual Studio since I don't have TFS. Is there another way to do this apart from TFS?
I ended up solving this issue by using MSTest and its Jenkins plugin. For those of you here with the same question I had, you can find some pretty good directions on how to get this working here (although I used the version you can find in VS 2012).
Related
In my project we are using QC to execute our test cases(QTP), moving forward we would be eliminating QC (for cost reasons).
As far as I explored MSBuild & Jenkins, they would be suitable.
But MSBuild will trigger the execution when a new build pushed to the repository. Also it will automatically test on the latest build.
Is there any other CI tool available to execute test cases through QTP?
I will be executing automation once in a release. Also we install our application by manual since it requires lots of configuration.
Take a look at HP Application Automation Tools.
This plugin basically replaces the need for QC, and is developed by HP.
Create a Jenkins job using this plugin on the same Jenkins installation used to build your code, then you can configure your job to run your tests as soon the code is available (e.g. on a nightly basis).
See here for a helpful guide on how to implement a simple Jenkins job using this plugin.
They also host the code on Github, which is very useful if you need to change the behavior of the plugin to suit your needs.
How can I get my TFS build to run my Specflow tests? I have the tests running in the local VS build, but they aren't being picked up in the TFS build. Please could someone post the steps I would take to have the build run the specflow tests, and report on the results. Thanks.
I guess specflow uses nunit tests. you need to set up nunit in build server
http://www.mytechfinds.com/articles/software-testing/6-test-automation/72-running-nunit-tests-from-team-foundation-server-2012-continuous-integration-build
If you use SpecRun, it will automatically take care of TFS Build integration. Installing SpecRun is just referring the NuGet package from the SpecFlow project.
It very easy using the TFS Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Team+Foundation+Server+Plugin) to check detect code changes committed to TFS, check out the code and compile it using Jenkins.
However I need to check a build artifact back into TFS after the build is complete. I can't see an option to do this.
Is there a way to commit artifacts tfs from jenkins build
This link might prove useful.
https://blogs.msdn.microsoft.com/visualstudioalm/2016/05/27/continuous-deploymentdelivery-with-jenkins-and-vs-team-services/
It's Developer Tools Blogs > Microsoft Application Lifecycle Management article which shows how to set up a Jenkins project that archives artifacts on TFS.
It's titled "Continuous deployment/delivery with Jenkins and VS Team Services" by Anand Gaurav...Really easy to follow and each step is carefully explained.
I'm looking to get TFS to build and deploy to our manual and auto TEST environments.
manual TEST - overnight updates
auto TEST - continuous integration
Once deployed, I want TFS to deploy and run a set of web-integration tests (SpecFlow / Selenium / nUnit) on another server (acting pseudo client of the website).
I then want TFS to collate the results of these tests and report back.
What's the best approach to get this working?
You'd normally change the build process template to do what you want. Adding steps to deploy could be one of these customozations. Have a look at the Build Release Deploy build template from the Microsoft ALM Rangers which includes steps to publish the build to another machine.
You can also make use of WebDeploy and add a set of publish profiles to your solution. Team Build will happily execute those steps during build, if told to do so.
In TFS you can use a Test Agent to then execute tests from a remote machine and have the results added to the test results for your build.
Seeing that you'd like to use NUnit ad SpecFlow I suggest using TFS 2012 in combination with Visual Studio 2012, which have built-in support for executing 3rd party based tests. You'll need to install the NUnit adapter and put the assemblies for it in source control so that Team Build will pick them up.
Does anyone have specific experiences with using TFS 2008/2010 AND Jenkins for Continuous Integration (CI)? We are trying to decide which CI server to use. Our team works exclusively in Microsoft .NET/Visual Studio 2010/C#. We have the following requirements:
Automatically build our web project on every checkin.
Run unit tests with each build.
Automatically deploy green builds to development and/or test environments.
Provide pretty reports.
Provide build/deployment notifications via email.
I realize that installing a tool won't necessarily give us this functionality out-of-the-box and that we will have to integrate with other tools like MSBuild to achieve this.
I'm looking for specific features that Jenkins has that TFS 2008/2010 does not or vice versa. Also which is easier to maintain, use, etc.
I would highly recommend using Jenkins - it will do all of your requirements out of the box apart from possibly #3, but if you can script your deployments then it can do that as well.
Here are some links to help you get your builds up and running:
Blog about doing .NET builds in Jenkins
Jenkins Windows installers
Installing the Jenkins master and slaves as Windows services
Disclaimer: I have no experience with TFS, but I think open solutions are nearly always more flexible and extensible (and cheaper !) than proprietary products.
Late to this game, but I have used both TFS 2010 and Jenkins for CI. TFS 2010 has minimum set of CI tools in it. However, when you want to create a CI pipeline, it's a completely different story while Jenkins can easily create the pipeline.
If you are looking at only CI for one build either one should work. However, when it comes to the entire pipeline, Jenkins is way to go. With TFS it can be done, but Jenkins is better choice.
Here's quick bullet points:
TFS:
With a build definition you can compile, execute tests, return changeset/workitems, send an email when a build is broken
natural integration with visual studio
extremely hard to create CI pipeline. Requires custom handler and extensive workflow work. Not as intuitive as creating a build definition.
Because of the 3rd bullet, it's not easy to maintain/customize/scale CI pipeline
Jenkins:
Need to create a msbuild config file for CI, which is not much pain comparing to creating CI pipeline using TFS. However, TFS gives better/easier tool to create a build definition. however, it is not bad creating config file for msbuild for a project.
Creating a CI pipeline is very easy. Just chain them using upstream/downstream jenkins job trigger and passing an artifact from previous job.
Since Jenkins is very flexible, it is easy to create a jenkins plugin to meet your own needs and provide it to opensource community :)
In summary, if you need complete automated build, test, and deployment system go with Jenkins. If you just need only build and test, TFS might give you an edge over Jenkins.
If you're using Team 2010-2012, there is no reason whatsoever to bring in Jenkins. Team has all the features you listed, and the build process is ridiculously flexible.
Note that if you are stuck on Team 2008 or earlier, you should seriously look at Jenkins -- 2008 and earlier are quite primitive and inflexible compared to 2010 or later.