We are using SpecFlow to apply BDD practices and Pickles to generate the documentation.
Our build looks like this:
Build
Deploy
Test (using VSTest)
Generate documentation (using Pickles)
The generate documentation step comes after the Test step, so we can include testresults to pickles. I want to use pickles as a build step, so I can manage the pickles version to use and additional options in each Solution.
The Visual Studio Build step in TFS2017 looks like this:
/t:DocumentationGeneration
/p:Pickles_Generate=True
/p:Pickles_DocumentationFormat=dhtml
/p:Pickles_ResultsFile="$(build.sourcesdirectory)/TestResults/*.trx"
/p:Pickles_OutputDirectory="$(build.artifactstagingdirectory)/PickledDocumentation"
[some more generic parameters that are not relevant here]
When I do the above, I get an "Illegal character in path" error during the build. It seems to be the * causing this error, when I replace it with /p:Pickles_ResultsFile="$(build.sourcesdirectory)/TestResults/testresults.trx" there is no problem at all, only the testresults are not found because the testresults are in the non predictive format:
USERNAME_SERVERNAME1234 2017-02-02 09_09_09.trx
I tried to look into generating a generic name for the .trx file, but VSTest does not support custom names. Did anyone else encounter this problem and is there a nice approach to solve it?
Pickles only supports the exact name of the test result file. There is an issue on the GitHub project site to allow wildcards. As the maintainer of Pickles, I will be happy to review any pull requests that address that issue.
At request: This is how I solved it. It is more a work around that works in my particular situation, but at the bottom I explain how I hope it will be fixed soon.
Create a Command Line build step right after test build step
as tool use Rename
as arguments use "$(build.sourcesdirectory)\TestResults\*.trx" TestResults.trx
Now you can use a Visual Studio Build step to generate pickles documentation (you need to have pickles as NuGet package in your project)
As Solution set $(Build.SourcesDirectory)/yourproject.csproj
As buildarguments set additional parameters:
/t:DocumentationGeneration
/p:Pickles_Generate=True
/p:Pickles_DocumentationFormat=dhtml
/p:Pickles_ResultsFile="$(build.sourcesdirectory)/TestResults/TestResults.trx"
/p:Pickles_ResultsFormat="vstest"
/p:Pickles_OutputDirectory="yourdestinationpath"
As I said, the renaming is a bit a work around, so I extended the code of pickles itself to enable wildcard support, and is now a pull request in the pickles repo, so hopefully it is implemented in the next release and you all can use it that way :).
Related
We use SpecFlow and NUnit in Visual Studio at work. Very useful, etc, etc. Once we've finished development the checked in code goes to a build server which uses Jenkins to build and run all the tests. Very helpful, etc.
However, there is an annoying hole: if you a/ use a step in SpecFlow that you don't define, a default step definition is used which marks the test as Inconclusive, and then b/ NUnit ignores Inconclusive tests (apparently MSTest fails them instead), and then c/ Jenkins doesn't detect a problem and passes the build. This means our build servers can pass tests that aren't even defined correctly.
So, can anyone EITHER:
- tell me how get SpecFlow to throw errors on missing steps? (can't find an option for it or anything on the web) OR:
- tell me how to get NUnit to treat Inconclusive tests as failing? (once more, can't find an option for it or anything on the web) OR:
- somehow get Jenkins to pick up the Inconclusive results and treat that as failing?
All suggestions will be considered, left-field ones included! Thank you.
There is a configuration option to change this behavior.
See here for the documentation. The configuration we're interested in here is missingOrPendingStepsOutcome.
The default setting is:
missingOrPendingStepsOutcome="Inconclusive"
We simply need to change (or add the setting) as follows:
missingOrPendingStepsOutcome="Error"
I am creating a build definition in visual studio 2013. As you know, while creating build definition we can also set the assembly (in "Automated Tests" section) which contains the unit test methods to run. I am able to create the definition till this point. Now, what I want is once the test run is over, I want to call an exe that will parse the Trx file generated as a result of test run and send the run results in an email.
Can anyone have an idea about where I can set the path to the exe and pass some arguments to it while creating build definition? If this cannot be done while creating definition, what else is the other way to achieve it?
Thanks!
What you ask involves multiple steps, especially if you want to pull the tool from version control (which is IMHO the best option).
See the ALM Rangers' Team Foundation Build Customization Guide for help and guidance on this topic.
Or you can just setup an email notification when the TFS Build completes. That should include the test run results, and a clickable link to go to the full build report.
In the TFS 2010 build definition window, under “Process” there are two required items. They are “Configurations to build” and “Projects to build”. Under projects to build, it will allow me to enter something like:
$/TeamProject/Area1/Area2/*
However, this doesn’t seem to do what I expect. The build fails because it’s looking for:
$/TeamProject/Area1/Area2/Sources/*
What I am trying to achieve by this is to build all the solutions held under this area. For example, I have:
$/TeamProject/Area1/Area2/Solution1/Solution1.sln
$/TeamProject/Area1/Area2/Solution2/Solution2.sln
$/TeamProject/Area1/Area2/Solution3/Solution3.sln
There are many more solutions than this, which is why I’m looking for a way to build all solutions under the specified path recursively. Is there a way to do this in TFS 2010?
You can modify the process template. Expand it with the Matching files (I don't have the exact naming now) activity. Add a parameter that passes the information you set in the build defintion to the MachingFiles actvity. Then pass into the build solution activity instead of the argument that you enter in the build definition the files that is found by the MatchingFiles activity.
Now add a dummy solution in the build definition for the solution to build (it is not used anymore).
See the blog post series on the build customization for more information on customizing the build process template.
FWIW,
I've got: "configurations to build" blank
and under "projects to build" I've added my solutions via the ellipsis button
I would setup mappings for
$/TeamProject/Area1/Area2/Solution1/
$/TeamProject/Area1/Area2/Solution2/
$/TeamProject/Area1/Area2/Solution3/
Then in the build definitions enter the three projects to build
$/TeamProject/Area1/Area2/Solution1/Solution1.sln
$/TeamProject/Area1/Area2/Solution2/Solution2.sln
$/TeamProject/Area1/Area2/Solution3/Solution3.sln
You can leave the configurations to build as blank, or if you want to do a certain build you can set it to (for example) something like Debug|Mixed Platforms (check your Configuration Manager... for the solutions you are building to see what is valid)
Alternatively, you can just map the following (depending on how much you have in this folder, if you have Solutions 4+ that you don't want to trigger builds on, don't do it at this level)
$/TeamProject/Area1/Area2
And have one solution which contains the Solution1, Solution2 and Solution3, and build that instead.
By default building your Solution1 which is mapped to
$/TeamProject/Area1/Area2
On a build agent with a working directory that is going to looks something like:
$(SystemDrive)\Builds\$(BuildAgentId)\$(BuildDefinitionPath)
You'll end up with it being build under
C:\Builds\1\Solution1\Binaries
C:\Builds\1\Solution1\Sources
C:\Builds\1\Solution1\TestResults
Which is why you want to make sure that your OutDir's etc are all correct and not hard coded!
If you have a look while building, you'll see the build agent populating the Sources folder, and it should (if configured correctly) put all outputs into the Binaries folder (and then copy them to the Drop Folder configured under Build Defaults in TFS.
I am playing with TFS 2010, and am trying to setup a build process that will have some custom steps.
These include things like, stopping/starting IIS, search and replace files etc... across environments.
I have tried to look for examples online and have not found anything clear and meaningful on how to just run a script or something over the source files. Looking at the default build process template (DefaultTemplate.xml) I cant make much sense of it.
How do I go about doing this ?
For info on customising the TFS2010 workflow build templates have a look at Ewald Hoffman's series. Start with Part 1 (archived here).
I should also mention that since it looks like you're doing deployment then you may want to break deployment automation away from build automation.
This is almost exactly what I'd say for this question (Split build and deplyment stages, investigate TFSDeployer). One additional element is more generic - for deployment tasks you can't find an easy integrated tool you should create a custom deployment script. You can call any script by adding an "InvokeProcess" step in your Build workflow. TFSDeployer also has locations where you can insert custom PowerShell Scripts. (If you don't like PowerShell you can have PowerShell or "InvokeProcess" call a different script engine.)
I have a vcproj file that includes a simple pre-build event along the lines of:
Helpertask.exe $(ProjectDir)
This works fine on developer PCs, but when the solution is built on our TFS 2008 build server under MSBuild, $(ProjectDir) is either blank or points to an unrelated folder on the server!
So far the best workaround I have managed is to hard code the developer and server paths instead:
if exist C:\DeveloperCode\MyProject HelperTask.exe C:\DeveloperCode\MyProject
if exist D:\BuildServerCode\MyProject HelperTask.exe D:\BuildServerCode\MyProject
This hack works in post-build steps but it doesn't work for a pre-build step (the Pre-build task now does nothing at all under MSBuild!)
Do you have any ideas for a fix or workaround? I have very little hair left!
$(MSBuildProjectDirectory) worked for me
I think your problem may be related to how items are initalized. An items include attribute is evaluated at the begining of a build. So if you depend on files that are created in the build process you must declare these as dynamic items. Dynamic items are those defined inside of a target, or by using the CreateItem task. I've detailed this on my blog MSBuild: Item and Property Evaluation.
Sayed Ibrahim Hashimi
My Book: Inside the Microsoft Build Engine : Using MSBuild and Team Foundation Build
I think the problem is that build server's workspace probably isn't initialized properly.
I just kept getting problems with this - I tried many different approaches but they all failed in mysterious ways.
Once $(ProjectDir) started behaving properly again, the pre-build step stopped executing the command (I added echo commands above and below it - they were both executed, but the program in between them was not. No errors or output of any kind were generated to indicate why it failed).
I don't know if this is a dodgy server of if MSBuild is having a laugh.
I've given up now. I gave the build server a big kick and have changed tack: We now run this tool offline (manually) and check in the results for the build server to use. So much for an automated build :-( If only MSBuild would run solutions in the same way as Visual Studio does - it's maddening that it sets up the environment completely differently (different paths coming out of the solution variables, ouptus redirected into different folders so you can't find them where they're supposed to be, etc)
I branched an existing project and $(ProjectDir) kept the old directory in the newly branched code. But that's because I had some compiling errors. Once every project in the solution compiled without errors, $(ProjectDir) changed to the correct path.
Carlos A Merighe