I have been working with Pex(IntelliTests) for some time now and I wondered if it is possible to create the tests via some sort of command(or .exe) and not through the IDE(VS2015) with right-clicking the function.
I have an automated process which builds my projects and further runs my tests. However if the IntelliTests are not generated anew for the new implementation they are rather useless.
This may seem like a basic question but unfortunately I could not find any information on the internet.
A command line for IntelliTest is not yet supported. If you would like to see it supported, kindly add your vote here: https://visualstudio.uservoice.com/forums/330519-team-services/suggestions/8623015-enable-intellitest-to-run-in-the-build-pipeline
Related
I am preparing for making a minor bug fix to bazel java code. Am working on a Linux distribution.
Following the instructions in https://bazel.build/contributing.html but I encounter problems with two of the test instructions:
In the section about "Compiling bazel" the third parapgraph state: "In addition to the Bazel binary, you might want to build the various tools Bazel uses. They are located in //src/java_tools/..., //src/objc_tools/... and //src/tools/... and their directories contain README files describing their respective utility." If I follow this the //src/tools/... fail because there is no xcrun command in the Linux environment I am using. I suppose this is MacOS platform specific tests?
The next paragraph instructs you to build a distribution package, that you then unpack in a new directory, and then do: "bazel test //src/... //third_party/ijar/...". I now get an error that windows.h is missing, which I suppose is Windows platform specific tests.
Some questions:
So is there an easy way to run tests only for the current platform?
Is the instructions good enough?
If the instructions should be updated, what is the best way to notify the ones managing that documentation page?
Thanks for your interest in contributing to Bazel! The bazel-dev mailing list is a better avenue for these questions.
The tests that you want to run largely depend on the changes you make, but when you make a pull request, the Bazel CI will run all of Bazel's tests to make sure that nothing breaks.
So is there an easy way to run tests only for the current platform?
It depends, and this is still a work in progress where we want to make Bazel more aware of platforms and toolchains without specifying additional flags.
In general, you don't need to modify or worry about the //src/*_tools packages unless you're making direct changes to them.
Is the instructions good enough?
The instructions will never be perfect, and we're always looking for ways to make it clearer and more concise.
If the instructions should be updated, what is the best way to notify the ones managing that documentation page?
Please file an issue on the GitHub repository or email the bazel-dev mailing list for further discussion.
Using Visual Studio Online I created a test plan for a program that was written by a different company that my company uses. We have a specific set of tests that need to be tested before we accept a new version of this program. So when I edit the test plan I would like to be able to manually select a build by typing in say version "1.0.1.195". Then when a newer version comes out I can just type in a newer version and retest using those same tests. However when I go to select a build TFS is filtering against my builds for my code. Is it possible to do what I'm asking using TFS?
EDIT
To answer a few of the questions in the comments I'll be a bit more descriptive of what I am doing. A 3rd party company made a program we use to test some hardware. Every now and then there is an update to that software. Since a few of us use this program to test out the hardware we need to know that the software can be installed with little to no down time while upgrading. So we came up with a small set of tests that we run the program through to make sure that we can test reliably. Those tests were written in a Word document, so I put them into MTM. Although I make some software that is related to this their software depends on mine. I've not had to update my code for some time now. My overall intention is to use MTM to document my testing of this program.
Do you want to store the version of the 3rd party component along with the test result of the test run it was tested with on TFS?
That would be nice. My ultimate end game is to put the results of said test back into that Word Document and make that available to those who don't have MTM installed (which is everyone). This way when a new version of the software is updated I can just go into MTM reset all my tests back to active update the version number and retest.
The build you set up in Microsoft Test Manager (MTM) defines where is the drop location containing your tests, not the application under test (it can be different if you build your tests using another build).
That's why you only can select one of your builds for your code.
What you are talking about is deployment.
That means you have to make sure the right version of the 3rd party program is deployed to the environment the tests are running on.
EDIT
What you need is a Test Configuration
Here you can find a is a good explanation how to create one: Test Configurations - specifying test platforms
The idea in your use case would be as following
(below I'm using terms described in the article mentioned above):
Create a Configuration Variable where you will store the current version of the 3rd party program
Create a Test Configuration and add this variable to it.
Set this Test Configuration as default test configuration for your test plan.
Be aware, if your test plan already contains test cases you will have to add this Test Configuration to each Test Case manually since only new added Test Cases get it assigned automatically
If you get a new version of the 3rd party program you will:
Add the new version of the program to the values allowed for the Configuration Variable
Open the Test Configuration you are using and update the program's version to the new one.
Reset your tests and run them.
Doing so you:
store all versions you have tested so far in the Configuration Variable since you add the new one instead of overwrite the old one, so you get a kind of history.
store the last version you have tested in the Test Configuration.
That should meet you needs.
Additional suggestion
(has nothing to do with your question but with your use case)
Consider describing tests within your Test Cases instead of creating a Word document.
Here is a good place to start reading: How to: Create a Manual Test Case
The benefits would be:
You can run your manual tests using Test Runner provided by MTM
Doing so you will have all steps you have done stored by the Test Result, you can add comments to each step when executing that, etc.
You can still export the test description to a Word document using this MTM add-on: Test Scribe.
Using this add-on you can also create a report of your test execution.
Additionally, if you are going to use MTM more in your daily job I would recommend you this free e-book Testing for Continuous Delivery with Visual Studio 2012
Is there any way of exporting Test Cases and Shared Steps from one project to another in TFS 2012 using database queries?
I have tried TFS Integration tool, but it did not work as I expected, so I was wondering if there is any way of doing this by connecting to SQL Server and exporting all items directly from and to the databases.
A few days ago I needed an Excel testcase export for a newer version of TFS (TFS 2013). Turns out that there is still no export to Excel function built-in to TFS (in contrast to E-Mail and Printing) and the tools became incompatible over time.
But I found this Revival-Tool that seems to work:
https://github.com/jorupp/ExportTestCases
Just wanted to point that out even if it may not be directly related to this topic with TFS2012.
You shouldn't work directly in the database, because it is not supported by Microsoft and you could harm it a lot.
I used the TFS Integration Platform tools a lot for migrating TFS2010 projects, but never used it for TFS2012. My experience with that were good and it shouldn't be a problem to migrate all Test Cases and Shared Steps into another TeamProject. If you have the same Work Item Type Definitions in both projects, you don't need to create field mappings.
Another option would be to use Excel as "Export" and "Import" cache, but you might loose some information, because not everything could be shown in Excel, e.g. Steps of a TestCase, the history, Work Item Links.
If you are skilled in programming using TFS API, you could write your own small migrator, but this could be a lot of effort.
All in all the TFS Integration Platform should be the easiest and best way, so what have been your expectations that have not been fulfilled?
You can use an alternative methods.
First export all Test cases using "Test case extractor."
Them import them in new project using "Test case import tool."
Adding to the point mentioned above, for exporting the test cases you might find the below link more useful:
http://tfstestcaseexporttoexcel.codeplex.com/
I have given it a try and found it very useful and easy to use.
We're using TFS 2008 for our source control. We're currently doing embedded work so our actual build process is using Analog Devices VisualDSP++ IDE. However, we do still use labels on TFS. What I had hoped was for the ability to have a script run when a label is applied. The idea here is to update version information in some data files. This would help ensure that every label programmatically updated that version string. Currently we have to do this manually.
I'm used to Subversion and Git to some extent and was hunting around for post-commit type hooks or something similar. TFS seems like a different beast.
I'm not a stranger to .NET and (so far as I know) could write a simple app that did the grunt work of updating our files and then applied the label programmatically but it seems like there's got to be an easier way.
Am I right? Is there some mechanism that I can attach a script/batch file when a label is applied in TFS?
No, sorry, there's no server-side event fired when labels are created/updated. I'd recommend using branches instead of labels. Then you can hook the CheckinEvent (filtered by the path of the branch[es] in question).
For more info on subscribing to CheckinEvent (including filters), check out Buck's blog + updates + links: http://blogs.msdn.com/buckh/archive/2006/09/29/checkinevent-path-filter.aspx
For a code sample that illustrates what your event listener might look like, check out Martin Woodward's "bunny": http://www.woodwardweb.com/vsts/behind_brian_th.html
For info on using branches instead of labels for code promotion, try http://tfsbranchingguideiii.codeplex.com/
I'm trying to work with MSBuild and TFS.
I've managed to create my own MSBuild script, that works great from the command-line. The script works with csproj files, and compiles, obfuscate, sign and copies everything that's needed.
However, looking at the documentation of TFS & Team Build, it appears that it expect solutions as the "input" for the script.
Also, I haven't found an easy/intuitive way of performing a "Get Latest Version" from the TFS as part of the script. I'm assuming that the Team Build automatically do a "Get Latest" on the solutions it's suppose to compile, but again - I don't (want to) work with solutions...
Any insights? any pointers? any links?
Team Build defines about 25 targets of its own. When you queue a Team Build, they are automatically run for you in the predefined order listed # MSDN. Don't modify this process. Instead, simply set a couple of these properties that determine how the tasks behave. For example, set <IncrementalGet> to "true" if you want ordinary Get behavior, or "false" if you want something closer to tf get /force.
As far as running your own MSBuild script, again this shouldn't be necessary. Start with the TFSBuild.proj file that's provided for you. It should only require minimal modifications to do everything you describe. Call your obfuscation & signing code by overriding a task like AfterCompile or AfterTest. Put your auto-deploy code in AfterDropBuild. Etc.
Even really complex scenarios are possible if you refactor appropriately. See past answers #1 #2.
As far as the actual compile, you're right that Team Build operates on solutions. I recommend giving it what it wants. I'll be the first to admit that *.sln files are ugly and largely undocumented, but at least you're offloading the work to a well tested & supported product.
If you really wanted to, you could give it a blank/dummy solution and override the CoreCompile task with your custom compiler logic. But this is really asking for trouble. At bare minimum, you lose all of Team Build's flexibility WRT building multiple platforms and flavors. More practically, you're bound to spend a lot of time debugging something that's designed to "just work" -- and there are no good MSBuild debuggers yet (that I know of). Not worth it, IMO.
BTW, the solution files do not affect the Get process. As you can see in the 1st link, the Get is done very early on, long before Team Build even reads the solution file(s). Apart from a few options like <IncrementalGet>, this is not controlled from MSBuild at all -- in particular, the paths to be downloaded are determined by the workspace mappings associated with the build definition. I.e., they are stored in the Team Build SQL database, not the filesystem, and managed with tools (like Team Explorer) that call the TFS webservice API.