I'm trying to setup our TFS 2015 server to run automated tests. I've got it running, but we need to run our tests in Debug mode (for various reasons I can't really adjust). The problem is that I can't seem to figure out a way to switch the configuration in the Test task.
The help that the task links to (here) says that it's as easy as selecting Platform and Configuration, but the problem is that those options don't exist for me (they exist under Reporting, but the help there suggests that they will simply compare the results to other builds with that configuration).
I've also investigated the vstest.console.exe parameters (help I found was this one) as well as modifying the runsettings file, but these only allow me to modify the platform.
Overall, my question is a)is there a reason why I don't see the Platform/configuration options in TFS, and b) given that I don't see them, how can I modify the configuration that the tests are running under?
If it helps, TFS is reporting the version as Version 14.95.25122.0, which corresponds to Update 2. I checked the logs for 2.1 and 3, but wasn't able to find anything that suggested that this was added in later versions (though I could be wrong).
UPDATE:
I've realized that I misread the Test documentation and that the Platform/Configuration options were always for reporting only.
My question is then if I can actually set this in the tests somehow.
Thank you very much for any help.
Assuming you want to compile your test project in Debug mode. You can add a VS Build step to specify the BuildConfiguration variable, and define debug for variable BuildConfiguration. Check the screenshots below:
Then in VS Test step, specify the Test Assembly as **\$(BuildConfiguration)\*test*.dll to test the assemble under Debug folder:
Related
I have a situation where my project needs to be tested in x64, but now I'm adding Coded UI tests which need to run in 32 bits.
I have previously used the multi-configuration option to spread work across multiple agents
The problem is that adding another variable for the test settings file will cause my build to run all categories in both x64 and x86 test settings and I want to be able to choose which categories use which settings, how can I do that?
No, you can't. Seems you want to achieve only build specifically configuration and platform such as Debug|x86 , Release|x64.
You cannot specify unique combinations of variables. For now, vNext Build will always build all permutations of the variables specified.
You could add a uservoice here, TFS admin and PM will kindly review your suggestion.
We use SpecFlow and NUnit in Visual Studio at work. Very useful, etc, etc. Once we've finished development the checked in code goes to a build server which uses Jenkins to build and run all the tests. Very helpful, etc.
However, there is an annoying hole: if you a/ use a step in SpecFlow that you don't define, a default step definition is used which marks the test as Inconclusive, and then b/ NUnit ignores Inconclusive tests (apparently MSTest fails them instead), and then c/ Jenkins doesn't detect a problem and passes the build. This means our build servers can pass tests that aren't even defined correctly.
So, can anyone EITHER:
- tell me how get SpecFlow to throw errors on missing steps? (can't find an option for it or anything on the web) OR:
- tell me how to get NUnit to treat Inconclusive tests as failing? (once more, can't find an option for it or anything on the web) OR:
- somehow get Jenkins to pick up the Inconclusive results and treat that as failing?
All suggestions will be considered, left-field ones included! Thank you.
There is a configuration option to change this behavior.
See here for the documentation. The configuration we're interested in here is missingOrPendingStepsOutcome.
The default setting is:
missingOrPendingStepsOutcome="Inconclusive"
We simply need to change (or add the setting) as follows:
missingOrPendingStepsOutcome="Error"
Using Visual Studio Online I created a test plan for a program that was written by a different company that my company uses. We have a specific set of tests that need to be tested before we accept a new version of this program. So when I edit the test plan I would like to be able to manually select a build by typing in say version "1.0.1.195". Then when a newer version comes out I can just type in a newer version and retest using those same tests. However when I go to select a build TFS is filtering against my builds for my code. Is it possible to do what I'm asking using TFS?
EDIT
To answer a few of the questions in the comments I'll be a bit more descriptive of what I am doing. A 3rd party company made a program we use to test some hardware. Every now and then there is an update to that software. Since a few of us use this program to test out the hardware we need to know that the software can be installed with little to no down time while upgrading. So we came up with a small set of tests that we run the program through to make sure that we can test reliably. Those tests were written in a Word document, so I put them into MTM. Although I make some software that is related to this their software depends on mine. I've not had to update my code for some time now. My overall intention is to use MTM to document my testing of this program.
Do you want to store the version of the 3rd party component along with the test result of the test run it was tested with on TFS?
That would be nice. My ultimate end game is to put the results of said test back into that Word Document and make that available to those who don't have MTM installed (which is everyone). This way when a new version of the software is updated I can just go into MTM reset all my tests back to active update the version number and retest.
The build you set up in Microsoft Test Manager (MTM) defines where is the drop location containing your tests, not the application under test (it can be different if you build your tests using another build).
That's why you only can select one of your builds for your code.
What you are talking about is deployment.
That means you have to make sure the right version of the 3rd party program is deployed to the environment the tests are running on.
EDIT
What you need is a Test Configuration
Here you can find a is a good explanation how to create one: Test Configurations - specifying test platforms
The idea in your use case would be as following
(below I'm using terms described in the article mentioned above):
Create a Configuration Variable where you will store the current version of the 3rd party program
Create a Test Configuration and add this variable to it.
Set this Test Configuration as default test configuration for your test plan.
Be aware, if your test plan already contains test cases you will have to add this Test Configuration to each Test Case manually since only new added Test Cases get it assigned automatically
If you get a new version of the 3rd party program you will:
Add the new version of the program to the values allowed for the Configuration Variable
Open the Test Configuration you are using and update the program's version to the new one.
Reset your tests and run them.
Doing so you:
store all versions you have tested so far in the Configuration Variable since you add the new one instead of overwrite the old one, so you get a kind of history.
store the last version you have tested in the Test Configuration.
That should meet you needs.
Additional suggestion
(has nothing to do with your question but with your use case)
Consider describing tests within your Test Cases instead of creating a Word document.
Here is a good place to start reading: How to: Create a Manual Test Case
The benefits would be:
You can run your manual tests using Test Runner provided by MTM
Doing so you will have all steps you have done stored by the Test Result, you can add comments to each step when executing that, etc.
You can still export the test description to a Word document using this MTM add-on: Test Scribe.
Using this add-on you can also create a report of your test execution.
Additionally, if you are going to use MTM more in your daily job I would recommend you this free e-book Testing for Continuous Delivery with Visual Studio 2012
I'm trying to work with MSBuild and TFS.
I've managed to create my own MSBuild script, that works great from the command-line. The script works with csproj files, and compiles, obfuscate, sign and copies everything that's needed.
However, looking at the documentation of TFS & Team Build, it appears that it expect solutions as the "input" for the script.
Also, I haven't found an easy/intuitive way of performing a "Get Latest Version" from the TFS as part of the script. I'm assuming that the Team Build automatically do a "Get Latest" on the solutions it's suppose to compile, but again - I don't (want to) work with solutions...
Any insights? any pointers? any links?
Team Build defines about 25 targets of its own. When you queue a Team Build, they are automatically run for you in the predefined order listed # MSDN. Don't modify this process. Instead, simply set a couple of these properties that determine how the tasks behave. For example, set <IncrementalGet> to "true" if you want ordinary Get behavior, or "false" if you want something closer to tf get /force.
As far as running your own MSBuild script, again this shouldn't be necessary. Start with the TFSBuild.proj file that's provided for you. It should only require minimal modifications to do everything you describe. Call your obfuscation & signing code by overriding a task like AfterCompile or AfterTest. Put your auto-deploy code in AfterDropBuild. Etc.
Even really complex scenarios are possible if you refactor appropriately. See past answers #1 #2.
As far as the actual compile, you're right that Team Build operates on solutions. I recommend giving it what it wants. I'll be the first to admit that *.sln files are ugly and largely undocumented, but at least you're offloading the work to a well tested & supported product.
If you really wanted to, you could give it a blank/dummy solution and override the CoreCompile task with your custom compiler logic. But this is really asking for trouble. At bare minimum, you lose all of Team Build's flexibility WRT building multiple platforms and flavors. More practically, you're bound to spend a lot of time debugging something that's designed to "just work" -- and there are no good MSBuild debuggers yet (that I know of). Not worth it, IMO.
BTW, the solution files do not affect the Get process. As you can see in the 1st link, the Get is done very early on, long before Team Build even reads the solution file(s). Apart from a few options like <IncrementalGet>, this is not controlled from MSBuild at all -- in particular, the paths to be downloaded are determined by the workspace mappings associated with the build definition. I.e., they are stored in the Team Build SQL database, not the filesystem, and managed with tools (like Team Explorer) that call the TFS webservice API.
I have a vcproj file that includes a simple pre-build event along the lines of:
Helpertask.exe $(ProjectDir)
This works fine on developer PCs, but when the solution is built on our TFS 2008 build server under MSBuild, $(ProjectDir) is either blank or points to an unrelated folder on the server!
So far the best workaround I have managed is to hard code the developer and server paths instead:
if exist C:\DeveloperCode\MyProject HelperTask.exe C:\DeveloperCode\MyProject
if exist D:\BuildServerCode\MyProject HelperTask.exe D:\BuildServerCode\MyProject
This hack works in post-build steps but it doesn't work for a pre-build step (the Pre-build task now does nothing at all under MSBuild!)
Do you have any ideas for a fix or workaround? I have very little hair left!
$(MSBuildProjectDirectory) worked for me
I think your problem may be related to how items are initalized. An items include attribute is evaluated at the begining of a build. So if you depend on files that are created in the build process you must declare these as dynamic items. Dynamic items are those defined inside of a target, or by using the CreateItem task. I've detailed this on my blog MSBuild: Item and Property Evaluation.
Sayed Ibrahim Hashimi
My Book: Inside the Microsoft Build Engine : Using MSBuild and Team Foundation Build
I think the problem is that build server's workspace probably isn't initialized properly.
I just kept getting problems with this - I tried many different approaches but they all failed in mysterious ways.
Once $(ProjectDir) started behaving properly again, the pre-build step stopped executing the command (I added echo commands above and below it - they were both executed, but the program in between them was not. No errors or output of any kind were generated to indicate why it failed).
I don't know if this is a dodgy server of if MSBuild is having a laugh.
I've given up now. I gave the build server a big kick and have changed tack: We now run this tool offline (manually) and check in the results for the build server to use. So much for an automated build :-( If only MSBuild would run solutions in the same way as Visual Studio does - it's maddening that it sets up the environment completely differently (different paths coming out of the solution variables, ouptus redirected into different folders so you can't find them where they're supposed to be, etc)
I branched an existing project and $(ProjectDir) kept the old directory in the newly branched code. But that's because I had some compiling errors. Once every project in the solution compiled without errors, $(ProjectDir) changed to the correct path.
Carlos A Merighe