I have an E2E Automation project in TFS. I'm using Protractor and Jasmine for the tests if that makes a difference.
I'm using a plugin for Protractor to generate reports and those reports are created locally and in the form of HTML files. I have a 'Reports' folder in my TFS project, so currently the only way to make those reports accessible to others is to manually right click the reports folder in Visual Studio, click add items and select the reports, update the folder, and then do a check in.
My question(s):
Is there a way to automate that
Is there a better way to do that
Is there a way to assign a TFS path instead of a local path to the reporter itself? (protractor-jasmine2-screenshot-reporter is the report plugin)
And a side question:
If those can update into TFS on every run, what's the best way to pull those files into some type of webserver / index page for anyone to view them without going through TFS?
You're approaching this all backwards. You want to run your Jasmine tests as part of your build/CI process and publish the results to the build that ran them, not by committing them as changes to your code repository.
You should be able to follow some instructions along these lines to accomplish it. Basically, use the "Visual Studio Test" task pointing to a Jasmine test file. It will run it using the Chutzpah test adapter and generate a test results file that the "Visual Studio Test" task can translate into a report to be attached to the build.
Related
We have been using TFS to run automated tests. The tests run either as part of a build or on demand from a Test Plan. We are using Specflow to generate the test scenarios.
For the first month that I was executing these tests, a .TRX file would be generated that contained some logs from specflow. I could also log my own trace logs using System.Diagnostics.Trace.WriteLine. All of that would show up in the .Trx file.
The .TRX file would also show the number of tests that passed, failed, or were inconclusive. If I opened the file in Visual Studio, it would show me a Test Run panel with details on each test and their results.
Recently, however, that stopped happening. The .TRX file is still created, but it is basically empty. It says that no tests were run and all of my Trace logs dont display. TFS still shows me the correct information when I am logged in to the console, but the attached .TRX file is always 2k, meaning its just the skeleton of an xml file.
Here are some details of how we run the tests:
We have a Test Plan, so the Tests run as part of a Release. The Release contains a single Run Visual Studio Test Task
The .runSettings file only contains the TestRunParameters sections. No other settings are configured.
We are using TFS 2017 Version 2.
I thought that maybe Specflow had something to do with it, but even a simple MSTest without Specflow wont log correctly.
Some answers have suggested add /Logger:trx to the Other Console Options, but that always leaves me with a Warning stating that the Release is running on multiple agents so that settings will be ignored. Our release isnt runnin on multiple agents, so I dont know why I see that warning.
This worked until recently, and I have tried to figure out what has changed. I tried creating a brand new build, new code base, and new tests to run but it doesnt seem to matter.
Let me know what additional information you need. Thanks
I have figured out how to fix the issue, but I dont understand what exactly happened so I hope someone can fill in the blanks.
There are multiple versions of VSTest.Console.Exe on the TFS server. The one that is used when I select Visual Studio 2017 as the version is
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
I dont know if that is correct or not, but that is what was being used. I noticed that locally I was using a different path.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
I was able to execute that version of vstest and generate TRX files locally. There is also ANOTHER vstest.console.exe located at
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\Extensions\TestPlatform\vstest.console.exe
I believe that last one is for VSTest V2? Im not entirely sure and would love someone to clarify that.
I believe which executable you run depends on the Test Platform Version setting. I normally had it set to Latest, but I changed it to Visual Studio 2017 while trying to figure out why it wouldnt log anymore.
TLDR - The Solution
You can, alternately select Specific Location, which allows you to provide a path to the executable. I set the Path to vstest.console.exe value to C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe and everything works fine.
The only thing I can think of is that the version of Vstest under Professional (the version of Visual Studio we are using) is different from the one under BuildTools? The test would run successfully either way, but the executable under Professional also generates the Trx file. Since no details are logged, its hard to tell what is going on.
Hopefully someone can explain what is going on here, but at least anyone who has this same issue has something to try.
We're using cake for defining our .NET builds, primarily so we can run the same build on developer stations as on the build server. TFS 2013 is our actual CI platform; the build workflow is effectively just a RunScript activity which invokes powershell and runs cake via its build.ps1 script.
The basic build is working well enough, and I'm on to having it generate reports (unit test results, coverage reports, etc). I'd like to have these reports appear in the build's Summary screen, but at the moment the only feedback the build gives me is the console output from cake under the build logs. The report files are being generated and dropped into the build's ./tst/ folder, but the contents of that folder aren't appearing anywhere in the build information.
How can I get test reports to be added to the build summary and/or information pages?
With the default build process template, when you build a test project, you will be able to get test result and code coverage (if you enable it) by default. So if you use VS Test Runner to run the tests, you can refer to the default process template.
If you need other tool to run the teats, you can add the InvokeProcess activity (execute the command line) to invoke the tool to test your project in build process template.
In addition, you can use the WriteCustomSummaryInformation activity in your workflow. The result is that you can display results, hyperlinks, and more on the build summary page.
Our plan is to use Team Foundation Build service to do automatic builds, then use the testing facility to automatically perform testing on the build server then release that build onto the application server.
So far we have
Team Foundation Server with TF Build Controller configured
Build server with win2012, Visual Studio 2013 and Build agent configured.
SQL Server with SQL 2013 installed
Application Server with Win2012 and .netframework installed
My question is what do I need to do to set up automatic builds, and to execute the unit test harness once compilation is successful.
Also the deployment target machine will initially be DEV, however we would like to quickly build for test env and prod etc.
This is what I got so far.
Build Controller (Already set up I believe)
Build Agent (Already installed on build server)
Build Process Template (Do I need to do anything with this. Is this what controls the whole lot)
Team Build Definition (I had a look at this, and it seems to use the build process template)
Drop Folder (I am assuming this is where the executables will be dropped into).
At the moment I have bits and pieces of info, what I would like to know is how this whole thing is hanging together. From the moment the developer wants to do the build to the moment that exe is placed into the DEVAPPSERV (Development application server).
Is anyone able to point me in the right direction or give a summary of what I need to make this happen?
Many thanks,
Dalibor
Install TFS Server (TFS Disk) Create a Team Project Collection and any desired Projects
Install TFS Controller + Agents onto a dedicated machine (TFS Disk) Configure only the build options if on a different machine to the TFS Server
Configure Build Controller to connect to a Specific Team Collection on your TFS Server
Install VS Premium or higher on build machine, if you want code coverage results for your tests
Add some code to TFS Source Control
Create a Build Definition using the default template.
Configure the build definition.
Set the working folder for the build, include only what you need as this will speed up the process
Point the definition to your .sln or proj file.
Ensure testing is enabled and that your test assembly names will match the regex used to identify test dll's i.e. name your test assemblies with the word test.
Set the trigger to be CI or what ever flavour of build you require i.e. gated build
Save the build definition
Trigger a manual build and debug any issues
you should have the basics done and a repeatable build created.
That should cover the basics, you may want to customise the build template (see Ewald Hoffman's guide for tips), you may want to narrow down your code coverage (look for runsettings file info).
If you follow these steps you should be able to get a basic build created and running from these, if you hit any issues you can come back and ask specific questions about a particular area
In order to do automatic builds you should check the CI build option ( under the trigger build option ) and third party automated testing can be run by executed by a post build script.
See the following TFS article about post build scripts.
http://msdn.microsoft.com/en-us/library/dn376353.aspx
We're using TFS Build and are having the TFS Build process run tests through a Test Controller on a Test Agent on a remote machine. We do this via a testsettings file in which we specify the test controller. This testsetting file is specified as an argument in the default TFS 2012 Build process template.
Everything works fine. However, our build takes about an hour. In the event that a test should fail because of an environmental issue (so in the event where it fails only on the test agent and not on our Dev machines), I'd like to be able to re-run the tests and diagnose the environmental issue that happened. However I can't seem to figure out how to do that WITHOUT re-running the entire build. Once the environmental problem is fixed, I'd like to be able to re-run the tests again to make sure everything is ok. I'd like to do all this without re-running the entire build. Ideally when I re-run my tests they should run in as similar a configuration as possible as to when TFS Build runs the tests.
Is there a way to do this? Is using MSTest command line an option to re-run the tests? If so, how do I know which arguments to pass to it? Ideally i'd want the same arguments TFS passed to the QTAgent process which ran the tests?
Create build specific, based on sequential workflow specific whose you delete from designer WWF all activities that don't need.
I suggest theses steps : 1. Select link 'Create Build Definition' 2.Create Process Template by copy pasting on Source Control, 3.Open your copied Workflow on designer 4.Delete all activities that you don't need. 5.Finish by associating this workflow to build definition (Select Process Tab)
The easiest way to prevent the whole build from running is to set "Clean Workspace" to None.
That means if you want to re-run the tests you can just queue the existing build definition manually, change to the Parameters tab and set Clean Workspace to None before click the "Queue" button.
In this case existing build outputs remain as well as source code files that have not changed since the most recent build (by performing a tf get without the /all switch).
It's probably not exactly what you want because if there were some check-ins since the most recent build the changed source files will be fetched and built.
But if it is not important for you to re-run the tests using exactly the same build output each time it could be a good solution for you since it usually reduces build time significantly.
One way for you to possibly do this is:
Open the build that had failed tests in Visual Studio
Scroll down to the test runs section.
Click on the failed test run. The Test Results window should open.
Select the failed tests you want to rerun.
click the "Run checked tests" button
Supposedly this should work: vstest to tfs logger. I can verify that the test are run and published to our tfs, but I could not see the test result on the tfs build page only using the link that is produced (the one with mtm as protocol)
I'm looking to get TFS to build and deploy to our manual and auto TEST environments.
manual TEST - overnight updates
auto TEST - continuous integration
Once deployed, I want TFS to deploy and run a set of web-integration tests (SpecFlow / Selenium / nUnit) on another server (acting pseudo client of the website).
I then want TFS to collate the results of these tests and report back.
What's the best approach to get this working?
You'd normally change the build process template to do what you want. Adding steps to deploy could be one of these customozations. Have a look at the Build Release Deploy build template from the Microsoft ALM Rangers which includes steps to publish the build to another machine.
You can also make use of WebDeploy and add a set of publish profiles to your solution. Team Build will happily execute those steps during build, if told to do so.
In TFS you can use a Test Agent to then execute tests from a remote machine and have the results added to the test results for your build.
Seeing that you'd like to use NUnit ad SpecFlow I suggest using TFS 2012 in combination with Visual Studio 2012, which have built-in support for executing 3rd party based tests. You'll need to install the NUnit adapter and put the assemblies for it in source control so that Team Build will pick them up.