We have TFS 2017 installed on our server and JMeter performance tests which i want to run through non-gui mode on TFS build agent.
I can't find way how to publish jmeter results to TFS?
There is no approved plugin for JMeter (at least none I could find) that incorporate your results into VSTS test results. You would need to dump your JMeter results locally, add a task that captures the output and puts it with the $(build.artifactstagingdirectory) path. You will then need to make sure that your final task is a Publish Build Artifacts. This will ensure that your JMeter results will get published as artifacts as part of the build.
Now the fun part, you obviously want to view the results so you are left with two options. The first is to create a web app that contacts VSTS via the API, pulls the JMeter artifact and displays the results as desired. There may actually be
Last option is to create a hub page for VSTS. This could query the build, get the JMeter artifacts and render them as desired.
These are the best options I could find for you at present. You can always request such a feature in the VSTS User Voice Forum.
There is not any build-in tool to integrate the JMeter with TFS for now.
You can have a try for Cloud-based Apache JMeter Load Test task. However it's cloud based, and need Visual Studio Team Services account.
There are detailed reports and analysis tool when using could based jmeter test such as this tutorial.
Please reference this article for detail steps : Running Apache JMeter based load tests in the cloud – how to
In your case, the simplest way is as virusstorm suggested using custom extensions to catch your test results and publish test results. If you are not familiar with this area, there are detail tutorials here. Moreover you could also ask help of some ALM company and team.
Related
I'm trying to get reporting working for Karate DSL, and it's proven a challenge because my team uses Circle CI instead of Jenkins. Cucumber reporting seems to only work for Jenkins.
I've had a look at this documentation, here:
https://github.com/intuit/karate/tree/master/karate-demo#example-report
https://github.com/jenkinsci/cucumber-reports-plugin
I was wondering if there is a circle friendly equivalent you could recommend? It'd be even better if the reports could be generated in the terminal. It's going to be a hard sell to convince my team to change CI tools just so I can implement a test framework.
Thanks!
Here's what I suggest:
If you follow the demo / doc instructions - you will get the HTML reports in say target/cucumber-html-reports, and this is "pure Maven and Java", no dependency on CircleCI at all so far.
Now all you need to do is somehow make these HTML reports accessible via the web. In Jenkins, there is an HTML Publisher Plugin. I am not familiar with CircleCI but a quick search suggests that there is a way to expose links to build artifacts.
Also note that when you follow the demo, Java JUnit XML reports would also be output to target/cucumber-reports. It looks like CircleCI has support for these which means that it should be able to derive the build pass/fail status and stats if configured right.
Also note that Karate now enables you to write custom reports: https://stackoverflow.com/a/66773839/143475
I've recently developed custom build runners (plugins) to TeamCity and Jenkins. The plugins enable the users to start automated load tests as part of the build process.
To give you some idea here's the Jenkins plugin page with a lot of description:
https://wiki.jenkins-ci.org/display/JENKINS/Apica+Loadtest+Plugin
Here are a couple of screenshots of the TeamCity plugin:
The setup GUI: https://i.imgsafe.org/5221a01.png
Build log: https://i.imgsafe.org/c93f7f9.png
Custom tab with load test summary: https://i.imgsafe.org/f5b6937.png
Setup validation example: https://i.imgsafe.org/52cc9a2.png
These Continuous Integration frameworks allow a high level of customisation for plugin development: the UI, the output, client and server code etc. can all be tailored.
I've received a question whether we can develop a similar build runner for MS Team Foundation Build Service. I've completely new to TFS, I've never used it for anything. I've read through a couple of tutorials on how to install and set up TFS but there seems to be very little material available regarding custom build runners. The closest I've got are the following pages:
https://msdn.microsoft.com/en-us/library/bb130146.aspx
http://blogs.msdn.com/b/jimlamb/archive/2010/02/12/how-to-create-a-custom-workflow-activity-for-tfs-build-2010.aspx
However, they don't provide any example on GUI extensions, validation, customisation etc.
Therefore I'd like to get the opinion of experienced TFS users before we get any more serious with the framework:
is it possible to develop plugins for TFS build?
what are the limitations? E.g. can I build a custom GUI with custom client/server actions?
can I include custom pages, show graphs etc?
Any advice is welcome.
Thanks for your help,
Andras
When it comes to TFS Build, you first need to know that there are two build systems: XAML Build and a new, now default, build system.
XAML Builds are based on Windows Workflow Foundation. You can create custom activities and add these to a Build Definition Template. XAML Build only run on Windows and extending them is not very easy.
The new Build system is based on Node, runs cross-platform and is very easily extendable. Microsoft has open sourced all the tasks they have for TFS Build (see GitHub for the repository).
Targeting the new build system means that you support on-premises installations of TFS 2015 and the cloud hosted version of TFS: Visual Studio Team Services (see visualstudio.com for more info).
The easiest way to get started is by creating a new VSTS account (free!), adding some code and running a build. If you have that working, you can start exploring the existing build tasks and learn what's possible. You can then easily create your own task and start experimenting.
One thing that might be of interest to you is that TFS/VSTS already support load testing. You can run a very simple load test with a couple of mouse clicks or configure more complex Web Tests and use these in TFS in combination with Application Insights. I'm not sure if that's what your customers are looking for but it's worth checking out (see Cloud-based Load Testing for more information)
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
I am working on an integration approach for integrating Rational (IBM Jazz) Quality Manager (RQM) and Requirement Composer (RRC) with TFS.
As the teams are moving from ClearCase to TFS, what we are looking at is still being able to manage the requirements in RRC and manage testing using RQM.
The flow will be something like:
Requirements are planned and detailed in RRC
Create work items in TFS connected to the Requirements in RRC
Create design and code using VS2010 and managing the version control in TFS
Creating test plans and test cases in RQM (connected to requirements in RRC)
Run test against builds in TFS
Publish test results in RQM against builds in TFS
Run reports in RRC, RQM and TFS that links up the items across the platforms.
I have started looking at TFS Integration platform. But shall like to have your guidance for an early resolution and better solution approach and how to achieve this.
UPDATE:
As I was not getting good response here, I put this on MSDN and I really got good response there. I am not deleting this post here, but give the link here:
http://social.msdn.microsoft.com/Forums/en-US/tfsintegration/thread/fcb20f89-6801-4ad1-8f46-3df68325ca8d
Please follow this.
I am using MSBuild for building our test project from command prompt which inturn uses nunit and gives the results. I want to schedule it nightly so that it automatically runs the msbuild and get the results. Is there any way I can do that?
I would recommend using a Continuous integration server for this. You can set up your continuous integration server to schedule builds and a number of other tasks. Personally, I use Team City from Jetbrains, but there are other options available, such as Cruise Control.net and Hudson.
What exactly do you want to do with the results? If you just need to run your build script, a simple scheduled task should do it.
The way my company solved the problem was to use Powershell. We have a scheduled task that executes a powershell script that completely automates the process using MSBuild and devenv.exe.
We've been able to do away with the expense of many other continuous integration tools with this process. We have even connected it to our svn server so that it can checkout the latest revision, build it, send out an email if there was a build error and send the output of the screen as a result log.
I realize this might not be simple answer you're looking for, but Powershell was really easy for me to learn, exceedingly customizable, and very powerful. I had doubts when my Director said we were moving this way, but it's been a delight.