We have been using TFS to run automated tests. The tests run either as part of a build or on demand from a Test Plan. We are using Specflow to generate the test scenarios.
For the first month that I was executing these tests, a .TRX file would be generated that contained some logs from specflow. I could also log my own trace logs using System.Diagnostics.Trace.WriteLine. All of that would show up in the .Trx file.
The .TRX file would also show the number of tests that passed, failed, or were inconclusive. If I opened the file in Visual Studio, it would show me a Test Run panel with details on each test and their results.
Recently, however, that stopped happening. The .TRX file is still created, but it is basically empty. It says that no tests were run and all of my Trace logs dont display. TFS still shows me the correct information when I am logged in to the console, but the attached .TRX file is always 2k, meaning its just the skeleton of an xml file.
Here are some details of how we run the tests:
We have a Test Plan, so the Tests run as part of a Release. The Release contains a single Run Visual Studio Test Task
The .runSettings file only contains the TestRunParameters sections. No other settings are configured.
We are using TFS 2017 Version 2.
I thought that maybe Specflow had something to do with it, but even a simple MSTest without Specflow wont log correctly.
Some answers have suggested add /Logger:trx to the Other Console Options, but that always leaves me with a Warning stating that the Release is running on multiple agents so that settings will be ignored. Our release isnt runnin on multiple agents, so I dont know why I see that warning.
This worked until recently, and I have tried to figure out what has changed. I tried creating a brand new build, new code base, and new tests to run but it doesnt seem to matter.
Let me know what additional information you need. Thanks
I have figured out how to fix the issue, but I dont understand what exactly happened so I hope someone can fill in the blanks.
There are multiple versions of VSTest.Console.Exe on the TFS server. The one that is used when I select Visual Studio 2017 as the version is
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
I dont know if that is correct or not, but that is what was being used. I noticed that locally I was using a different path.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
I was able to execute that version of vstest and generate TRX files locally. There is also ANOTHER vstest.console.exe located at
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\Extensions\TestPlatform\vstest.console.exe
I believe that last one is for VSTest V2? Im not entirely sure and would love someone to clarify that.
I believe which executable you run depends on the Test Platform Version setting. I normally had it set to Latest, but I changed it to Visual Studio 2017 while trying to figure out why it wouldnt log anymore.
TLDR - The Solution
You can, alternately select Specific Location, which allows you to provide a path to the executable. I set the Path to vstest.console.exe value to C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe and everything works fine.
The only thing I can think of is that the version of Vstest under Professional (the version of Visual Studio we are using) is different from the one under BuildTools? The test would run successfully either way, but the executable under Professional also generates the Trx file. Since no details are logged, its hard to tell what is going on.
Hopefully someone can explain what is going on here, but at least anyone who has this same issue has something to try.
Related
In TFS Test Task I know that I can manually set the location of vstest.console.exe via:
this is less than an ideal solution though since, as I update my build server, this location may become invalid. The problem that I have run into is that if I select 'Version' my build server ends up using the vstest.console at this location:
C:\Program Files (x86)\Microsoft Visual Studio\2017\ SQL\Common7\IDE\CommonExtensions\Microsoft\TestWindow
I am guessing that this SQL Folder is the SQL Server Data Tools folder that is installed on the build server. This folder location isn't generally a problem, vstest runs my unit tests and reports back on them, however, it doesn't produce a code coverage report. I initially thought that this was due to an improper configuration of the builds, however after verifying that the build was setup correctly and that the server had all of the necessary dependencies I started looking for other causes. I opened up the .trx file from one of the builds and noticed that the DataCollector logged a message that it couldn't find CodeCoverage.exe. After some more troubleshooting, I pointed my build to the location of the vstest.console shown in the image above and everything (tests, code coverage) ran fine. This leads me to believe that vstest expects the path:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Dynamic Code Coverage Tools
to exist relative to where it is run from and that directory (Dynamic Code Coverage Tools) does not exist inside the SQL folder.
Is there a way to change the path that TFS uses when the 'Version' checkbox is checked?
VsTest task uses vswhere.exe in order to determine the latest version of VS installed in the machine. Can you try to run vswhwere.exe and check which version of VS is installed at C:\Program Files (x86)\Microsoft Visual Studio\2017\ SQL folder and is it higher than other versions, also check whether its an enterprise installation or not. As code coverage is provided as part enterprise installation only.
You will be able to find vswhere.exe in agent working folder as per VSTest task version eg.
C:\Temp\Agents\127Agent_work_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.3.19\vswhere.exe
I have an E2E Automation project in TFS. I'm using Protractor and Jasmine for the tests if that makes a difference.
I'm using a plugin for Protractor to generate reports and those reports are created locally and in the form of HTML files. I have a 'Reports' folder in my TFS project, so currently the only way to make those reports accessible to others is to manually right click the reports folder in Visual Studio, click add items and select the reports, update the folder, and then do a check in.
My question(s):
Is there a way to automate that
Is there a better way to do that
Is there a way to assign a TFS path instead of a local path to the reporter itself? (protractor-jasmine2-screenshot-reporter is the report plugin)
And a side question:
If those can update into TFS on every run, what's the best way to pull those files into some type of webserver / index page for anyone to view them without going through TFS?
You're approaching this all backwards. You want to run your Jasmine tests as part of your build/CI process and publish the results to the build that ran them, not by committing them as changes to your code repository.
You should be able to follow some instructions along these lines to accomplish it. Basically, use the "Visual Studio Test" task pointing to a Jasmine test file. It will run it using the Chutzpah test adapter and generate a test results file that the "Visual Studio Test" task can translate into a report to be attached to the build.
I’ve been reading some articles (this one in particular: http://speaksql.wordpress.com/2013/06/07/a-journey-to-db-deployment-automaton-ssis-build-using-msbuild/) about using msbuild.exe to build and deploy an SSIS package (.ispac). I had no problem with that from my computer which has all the required assemblies, and only using the msbuild.exe command.
Once I tried to use the TFS Build Server I had some problems. First I realized that using a project with msbuild.exe (SSIS.MSBuild.proj) was not recommended with TFS 2013 since it was used with TFS 2008 and 2010. Anyway, I just wanted to make it work and it sounded plausible, but I had to use the TFSBuild.proj. That was not a problem, but my next problem was that the project I was being using (Microsoft SQL Server Community Samples: Integration Services) references a SQL Server assembly (Microsoft.SqlServer.ManagedDTS) which is not installed on my build server.
Then I realized that even if I managed to install that assembly on the gac, or referenced it on a relative path I would have a bigger problem next, I am using custom activities on my packages which I need to install using gacutil.exe on the host server, and I was wondering how to install, remotely, those dll.
That’s when I started to lose faith, and here I am, is there an “easy” way to implement continuous integration for SSIS packages without installing third party tools (http://remotegacutil.codeplex.com/ for example), and adding missing assemblies to the gac of a build server?
Did any of you have a similar issue? Did you solve it? How?
Thanks!
Use devenv.exe to build the ispac. Add an invoke process task and call out to devenv.
Add an Invoke Process to the Build Template, the one above shows a Sequence container, which assigns the path to the correct version of DeVenv to call, VS2010 /12 /13 etc. this is shown as hard coded but can be put into an argument, so it can be populated in individual build definitions. the one you can't see clearly is "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\devenv.com"
Next is the invoke Process, this call out to the specified devenv.exe and passes arguments used to build the ISPAC file.
The arguments are passed in localProject would get you the actual project you wanted to build in previous versions, you may have to do something different for 2013, due to the changes to the new templates.
this gives a TFS2012 way of doing it, as i say you may have to do something different to get the project that you want to build under 2013, but the build will run and an .ISPAC file will be generated.
at this point i would deploy the ISPAC using powershell, you may want to add additional scripts to the powershell calls to handle creating the SSIS Catalog and scheduling of the job.
deploying with Powershell can be found here Deploying ISPAC's with Powershell
I am new to TFS and want to integrate OpenCover with TFS. If any has done this please help!
This question is rather old but maybe you are still interested. With current Version of TFS (2015 Update 2) this is now possible as a "vsts Extension".
See here for details: https://github.com/RobertK66/vsts-opencover
Since the answer doesn't specify the version of TFS, here is an answer for 2015/2017.
OpenCover can be run from TFS using the Powershell build step. You need to get the contents of the OpenCover NuGet package onto TFS and run OpenCover.console.exe from there.
Since TFS doesn't support the format produced by OpenCover, you need to take one additional step and convert the results to Cobertura format. It's possible using the OpenCoverToCoberturaConverter NuGet package.
I've described the whole process in much more detail on my blog:
http://codewithstyle.info/setting-up-coverage-reports-on-vsts-with-opencover/
OpenCover is just a console application so you can just modify your scripts to get OpenCover to run your unit tests.
I haven't used TFS for several years and it has changed since then however this blog post should help
To incorporate coverage measurement of OpenCover the build process of TFS (second half)
The original is in Japanese but if you are familiar with TFS then the screens will probably be obvious.
OpenCover also comes with an MSBuild task that may help you with your integration.
I've just integrated opencover with TFS Build, to generate a xml with the results that will be processed by sonar:
Created a RunCoverage.cmd at the root of my source folder
Installed / copied in TFS Servers the Opencover binaries and added into Path(must restart TFS Services to enable TFS to see it).
Created a new actitivity in the build definition, that is, editing the build template (before sonar execution activity) to run the cmd:
Running command line statements from TFS custom activity
http://msdn.microsoft.com/en-us/library/gg265783.aspx
There is the cmd contents:
#REM #echo suppresses command line. ^ is line continuation character
#ECHO Executing OpenCover...
OpenCover.Console.exe -register:user ^
-target:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" ^
-targetargs:"/testcontainer:%~dp0test.dll /usestderr /resultsfile:%~dp0MSTestsResults.trx" ^
-output:%~dp0opencovertests.xml
But i'm facing three issues (that are related with my concrete implementation, but you may face them):
The Tests are runned twice (one for template itself, and other for OpenCover)
The MsTest.exe in the TFS Servers is not in the same path, so when the Controller asigns an agent (if the match is done by tags) then if the agent executing the build is in a TFS Server that does not have the MSTest in the right path it will fail.
How to inject in cmd the corresponding test runner (MsTest, XUnit, Nunit, etc...) depending on the test project
Hope that it helps! (and someone can help me ;-)
I am attempting to run UI tests (using Windows Automation API) on a WPF application, on a CI basis. I have associated my solution with a .testsettings file and linked my test .dll to the build process. All seems to be well.
When I queue a new build, in TFS's build log I can see the command line arguments passed to MSTest (2010) on the build machine - it appears that the .testsettings file is loaded correctly and that the correct test .dll is passed to MSTest.exe.
However, it seems to be "stuck" after the Executing tests message - both when running through the build and by running it directly through the build machine. In addition, I have specified a startup script to run before the tests, in the .testsettings file (it copies the application's binaries locally so the test can start the application and run the applicable tests). This script doesn't seem to be executed (the directory to which the files are supposed to be copied is empty).
Has anyone seen a similar problem and knows how to deal with it?
As Mike said in the comments, the issue was actually a bug in VS 2010 that made it impossible to run tests through the 2010 IDE if you have VS 2012 installed side by side. Unfortunately, I was unable to install VS 2010 SP1 on the build machine, but it is also possible to solve the issue by adding the /noisolation argument to the standard command line syntax (this runs the tests through the MSTest process, which somehow solves the issue).
However, since you cannot define additional arguments to be passed to MSTest through TFS's integrated automated testing feature, I wrote my own application that invokes as a scheduled task, runs the tests and sends an email containing an HTML report (I used trx2html for that -- note that if you're using VS2012, you'll need the beta 0.7 version, as there's been a change in the .trx format since VS 2010).