Chutzpah integration with TFS 2012 - tfs

My team is trying to integrate Chutzpah into the TFS 2012 build process. We used this blog post as our starting point.
At a high level, the practical issue is that the Visual Studio Test Runner in the build agent context simply isn't finding the Chutzpah hooks. So while we can define **\*.js as a test source, without the Chutzpah bootstrapper actually being found and initialized, the test runner doesn't do anything with these files.
At a more detailed level, we are getting three concerning messages when we check the logs for loading the custom assemblies for the build controller:
Summary: There were 0 failures, 2 errors and 1 warnings loading custom activities and services.
Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=2.2.0.171, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.
Error: API restriction: The assembly '...\AppData\Local\Temp\VSTFSBuild\8c8e9402-1169-4782-99a9-ce42f83be8f0\A1288811191\Chutzpah\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll' has already loaded from a different location. It cannot be loaded from a new location within the same appdomain.
Warning: Could not load file or assembly '...\AppData\Local\Temp\VSTFSBuild\8c8e9402-1169-4782-99a9-ce42f83be8f0\A1288811191\Chutzpah\phantomjs.exe' or one of its dependencies. The module was expected to contain an assembly manifest.
Other than this information, we're more or less stuck. I'd love to hear from someone who has actually got Chutzpah running on a standalone 2012 build server so we can compare configurations.

Error: API restriction: The assembly '....
Indicates that TFS is finding a dll in two different locations. You can change the pattern to Check and see if you have \*\*test.dll as the value set for any test assemblies and change it to *test*.dll. This will prevent it loading it multiple times.
The other error might be because the test projects are not being rebuilt.
Try re-building the test projects.
I hope this helps.

Actually Software Carpenter has a point, I believe what may be happening is your test spec for ordinary unit tests may be test.dll (or something similar) this means Microsoft.VisualStudio.TestPlatform.ObjectModel.dll is loaded as a unit test dll, then TFS tries loading it again when trying to run the chutzpah tests.
Try disable the normal unit test spec and see if it helps, if it does then change your spec to be something else (*test.dll) that doesn't incude TestPlatform.ObjectModel.dll.
Source: I just had the same error when trying to build a project with Test in its name.

This discussion is also taking place on Codeplex. Maybe some help can be provided there.
Javascript Unit Tests on Team Foundation Service with Chutzpah

Related

TFS: only SOME tests are discovered in the same assembly

Using the very latest TFS 2017 release 2 (about to upgrade to 3 by the look of it), on-prem. Tests are MSTest. I recently consolidated all our tests into one assembly, which are usually distributed to 6 VMs to process in parallel (have been using only 2 lately for unrelated reasons, a problem which I am currently engaged with MS in solving). The tests not being discovered are not included in the total that shows in the log/console during the Run Functional Tests step of the build. It should be ~1400 total tests, but it shows only 896. As far as I can tell, the tests not being properly discovered are the ones that were in the assemblies that got consolidated out of existence. My method of consolidating was basically to move the code files (.cs) from those assemblies to the single assembly we have now and adjust the namespaces so that all the tests are in the same namespace. No other code changes.
So how can it find only some tests, when all have appropriate attributes (the build definition is set to run only tests with a category of "Automated", which all of these tests have) and all in fact are in the very same namespace? I am at a loss.

How to adjust test configuration in TFS 2015 Visual Studio Test task

I'm trying to setup our TFS 2015 server to run automated tests. I've got it running, but we need to run our tests in Debug mode (for various reasons I can't really adjust). The problem is that I can't seem to figure out a way to switch the configuration in the Test task.
The help that the task links to (here) says that it's as easy as selecting Platform and Configuration, but the problem is that those options don't exist for me (they exist under Reporting, but the help there suggests that they will simply compare the results to other builds with that configuration).
I've also investigated the vstest.console.exe parameters (help I found was this one) as well as modifying the runsettings file, but these only allow me to modify the platform.
Overall, my question is a)is there a reason why I don't see the Platform/configuration options in TFS, and b) given that I don't see them, how can I modify the configuration that the tests are running under?
If it helps, TFS is reporting the version as Version 14.95.25122.0, which corresponds to Update 2. I checked the logs for 2.1 and 3, but wasn't able to find anything that suggested that this was added in later versions (though I could be wrong).
UPDATE:
I've realized that I misread the Test documentation and that the Platform/Configuration options were always for reporting only.
My question is then if I can actually set this in the tests somehow.
Thank you very much for any help.
Assuming you want to compile your test project in Debug mode. You can add a VS Build step to specify the BuildConfiguration variable, and define debug for variable BuildConfiguration. Check the screenshots below:
Then in VS Test step, specify the Test Assembly as **\$(BuildConfiguration)\*test*.dll to test the assemble under Debug folder:

Irregular build errors in TFS 2015 (missing assembly)

We have set up TFS (2015) with a build server and have several solutions that are built (some manually, others automatically).
Sometimes a build will inexplicably fail, with a lot of errors stating that assemblies are missing, like so:
The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
To fix it I simply need to queue the build again. No edits, no check-ins, just "queue new build", and it builds successfully.
It can happen on manual builds, on automatic builds, on a single build or when building multiple solutions. I can't see any recognizable pattern to why this happens. I don't think it is a nuget issue, because we see different errors when the build server fails at downloading nuget packages (also, is nuget even used for downloading System.Runtime? I assume that library is readily available on the server?)
As I said, it is easy to fix this, but we're using continous integration and automatic deployment (Octopus) to streamline our release cycle, and these "fake" errors are damn annoying when the build error buzzer starts ringing.
Try to set 'MSBuild Arguments' to /m:1 to force MSBuild to use a single process for all projects, to see whether it is helpful.
The issue was, as might be expected, human error. All available build agents were added to the pool for this particular solution, and one of those agents didn't have .Net Framework 4.5.2 installed.The agent was removed from the pool, and if the feedback I got from the sysadmins is correct then the errors should stop occurring now.

One build definition won't generate fakes assemblies, another one does

Introduction
I have a problem with Team Foundation Server Express 2013 on my machine. I have two build definitions on the same controller and agent, both of which run on the same server and the same environment as well.
It should be noted that I already looked at the "similar questions" without any luck. This is clearly not related to the same root cause, and the symptoms are slightly different too.
One of them is a gated check-in build definition, which just compiles everything when commiting to the development branch.
Another is a scheduled build definition, which runs every saturday at 3 AM, building any changes that may have been committed to the main branch since last time.
The gated build definition has a process (which only has minor changes for not running tests and just compiling the code) based on the TfvcTemplate.12.xaml template.
The scheduled build definition process is based on some Azure build definition template that might come from an older version of Visual Studio, possibly based on some Azure template, or maybe the TfvcContinuousDeployment.12.xaml template.
The issue
My gated build definition runs just as expected, without issues. It compiles the full solution, and only passes if the compilation succeeds.
The shceduled build definition however fails compiling (even before it reaches the point where it runs the unit tests). The error I see is as follows.
Obviously this is due to missing fakes assemblies. I tried taking the assemblies and checking them in (which I would rather avoid), only to find that this build definition runs just fine, but not the gated one which ran just fine before.
I thought about just running fakes.exe in the build template to just generate them manually before compiling, but in my initial tests (to see if this theory would even work), it won't even run in the commandline, and outputs some errors and warnings that I don't understand (but are probably not relevant anyway, since I might be running fakes.exe with improper arguments).
Updates
Update #1
It should be noted that I have Visual Studio 2013 Ultimate installed on my build server as well. Both this (and TFS 2013 Express) have Update 3 installed, and the server is fully updated.
I ended up abandoning fakes all together, and implementing Moq instead. Works a lot better, and forces me to abandon shims or moles, which are often considered bad practise anyway.

FSharp FAKE - Tracing Process Output

I have a FAKE task where I am calling Visual Studio to build a setup project. I have been playing around with ExecProcess and ExecProcessRedirected, but am unable to get tracing to work. In other words, I want to log all of the build output messages from VS as it runs.
My honest opinion is: don't use VS during a build. You'll end up installing VS on CI servers and everything will become worse. Try to find a different way. You can use WiX which has already a FAKE task or other installer tools which don't require VS.

Resources