We currently have coded UI regression test suite (UI function test) which runs perfectly using MTM on test anent/controller.
As part of CI our team is using Jenkins. Here we want to integrate coded UI test with Jenkins. Can you please tell me how can I configure this?
Is there a way Jenkins will access the TFS build of my coded UI test project which will have test run setting file (with info of test controller and agent to run test)?
I don't want to run coded UI tests on Jenkins server (but on test agent) as it will need to have a active desktop session.
A great example of running UI with Jenkins is executing selenium tests. This approach suggests you to run browser and simulate user actions. The same you can do with your UI and basically this does not require any additional configuration steps except slave configuration. You need to use Slave Configuration via Java Web Start as only this gives you ability to interact with Desktop:
If you need display interaction (e.g. for GUI tests) on Windows and you have a dedicated (virtual) test machine, this is a suitable option.
Other slave configuration ways may just not run your UI.
Related
I have several automated tests that are associated to test cases in TFS 2017. These tests are running through as part of a functional tests step as a part of our build and they're running fine and I can see them running on the test machines. The physical browsers are there and navigating.
I'm attempting to run automated tests through TFS test hub using the new 2.* version build agents. This will allow us to run individual tests or tests plans.
However, I can't figure out a way to run tests on these agents where they're not running headless. They run headless fine, but I need to be able to see them run.
I read somewhere in Microsoft documentation that the agent must run as interactive instead of a service for UI tests. However that means the agent is running as a version 1.* which is not supported by the TFS test hub test runs.
Does anyone know a way to use the test hub to run test on physical browsers instead of headless?
The new agent 2.0 can also be configured to run in interactive mode, please see Deploy an agent on Windows for details.
Just for your information:
If you are running UI tests such as CodeUI or Selenium on physical browsers such as IE, Firefox, or Chrome, the agent on the machines must be running in interactive mode and not as a service.
If you are running UI tests such as coded UI or Selenium tests, the agent on
the test machines must be running in interactive mode, not as a
service, to allow the agent to launch a web browser. If you are using
a headless browser such as PhantomJS, the agent can be run as a
service or in interactive mode. See Build and Release Agents, Deploy
an agent on Windows, and Agent pools and queues.
Please see Run automated tests from test plans in the Test hub for details.
I have few Coded UI tests which are not in TFS. I was running these tests as ordered test from the CommandLine using the command:
vstest.console OrderedTest2.orderedtest
Now, I want to run this through Jenkins but i am seeing the below error:
Error Message:
Error calling Initialization method for test class SAClientAutomation.testsuites.SABVTSuite: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
I have installed Test Agent and Test Controller but i don't what to next.
The code is not in TFS. Is TFS & MTM(Microsoft Test Manager) compulsory to achieve this?
Yes, TFS & MTM can run Coded UI tests. If you use TFS 2015.1 and previous versions, you need MTM to run the Coded UI tests, check: https://msdn.microsoft.com/en-us/library/ff472576(v=vs.120).aspx.
If you use TFS 2015.2 and later versions, you can use Run Functional Tests task directly in TFS build definition to run Coded UI tests. Check https://blogs.msdn.microsoft.com/visualstudioalm/2016/03/31/executing-automated-tests-in-build-vnext-using-test-plan-test-suites/.
In order to run coded UI tests in TFS 2015.2 and later, you need to deploy build agent and test agent in Interactive Process:
https://www.visualstudio.com/en-us/docs/build/actions/agents/v1-windows
https://www.visualstudio.com/en-us/docs/build/steps/test/visual-studio-test-agent-deployment
Selenium test (testNg) with mvn test from jenkins ,
I cant view the action performed on the browser, it is possible to view the real browser opened and performing action?
I suggest using SeleniumGrid.
You can then let the test run on a node, where you can see all the browser actions. For demonstration purposes, distributed and independent testing I did similar with teamcity for a regression-test project.
Basically there are two steps to setup the environment:
Start the grid for example as background-service.
Start a node which is running within a logged on user context (also as Service or autorun)
optional: Start another node for running tests in background
In your tests the creation of the the WebDriver instance changes like this:
WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capability);
Here a link for more details to setup the Grid2: https://github.com/SeleniumHQ/selenium/wiki/Grid2
No. In typical setup, tests from Jenkins run on randomly available node. You don't know which node is available at a given point. Also typically Jenkins nodes tend to run headless browsers (like phantomjs) since they don't have displays.
Jenkins tests are meant to be run without any manual intervention, like on nightly basis for example. You develop tests locally, see how they run in a real browser and then push it to your main suite so that Jenkins can run it for you. If you want to run tests from Jenkins and view test running live then look for paid solutions like saucelabs or browserstack.
You can using BrowserStack. We have recently released a Plugin that allows you to view your test results right inside Jenkins. It's a paid tool of course :(
In Windows you should not install jenkins as windows application (because in this recent version, Microsoft decided to give services their own hidden desktop even you enable the functionality "interact with desktop" in jenkins service), you may have to deploy it from a war file as follows:
1) Download jenkins.war from Jenkins official site
2) Deploy it by the command prompt : java -jar {directoryOfJenkinsFile}/jenkins.war
3) Now you can access jenkins administration on http:// localhost:8080
Hope that helps you !
I would say that my problem rather lack of information and I need some confirmation than a real problem. It seems somebody else had similar question question.
I put together a machine (Windows Server 2012R2) for POC reasons where a Jenkins installed and it executes Selenium UI tests using nunit. The nunit tests are generated by Specflow.
I could do:
install jenkins
jenkins run by a valid user not by Service account
set up jenkins properly
it can pull the source code from TFS-GIT
it can compile the C# solution
it can execute the test project
the test results are correct
Selenium plugin installed on Jenkins but I don't think it is used in this case because the text execution is about executing nunit and it deals with everything else.
At the moment I don't need the capability to delegate test execution to other Jenkins slaves or machines because the Jenkins does have only one compile task. Compiling, executing and test running can go parallel, the machine able to deal with it.
But, when I log in the server where the Jenkins runs and I watch what happens during CI build (compile and test execution) I can't see that the browser (Firefox) starts, however, the test results and the logs show that a browser was executed.
What I did so far:
jenkins runs as service, the account is an existing account
If I remote to the machine with the account which is set up for the service, then I can't see the browser will be executed, however, the log shows that something had happened.
My question is that, what the hack is happening when my tests are executed by Jenkins? If I execute the command which is used by Jenkins from console on the same machine then I can see that Firefox starts, does what is programmed in the tests and the results are in the result.xml. Can I accept the result as valid result? Can I somehow set up Jenkins the way the browser really executed (I can believe it when I see it :) )?
I think this is because you run Jenkins as a service. Services do not show up in desktop. Workaround is to run Jenkins or slave from CMD.
Jenkins windows slave service does not interact with desktop
I am using Jenkins for my project CI. And several automation test cases developed by Coded UI(C# language) will be run when the new build is deployed. I created a job on the master(windows) that MSTest those test cases(note: with MSTest under cmd, test cases can be run successfully) but saw the below issue:
Error calling Initialization method for test class QuickUI.SmokeTests: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=159018)
Therefore,
1) Need I use Jenkins slave to run test cases?
2) If so, how to config since the dlls needs to be copied to slave firstly?
3) Is there any account or jenkins service configuration needs to be taken care?
Thanks in advance.
Sounds like you need to get mstest setup on your build machine. The easiest way is to use test agents (http://www.microsoft.com/en-us/download/details.aspx?id=38186) when installing chose test agents.
(below is copied from my answer in Coded UI build server integration process)
You going to want to put your CodedUI tests inside an orderedTest. Right Click on the solution -> add -> orderedTest.
You going to want to install https://wiki.jenkins-ci.org/display/JENKINS/MSTestRunner+Plugin it's not a requirement, but it makes working with mstest in jenkins a little easier.
At this point you just need to configure the plugin in jenkins to run your orderedTest. It will need you to point to mstest and the location of your orderedtest.
1) Recommnd to run test on slave
2) To run GUI test, don't run the slave as windows service
3) Theoratically you only need the slave agent running. However the windows account logged in should have access to all the resources that your tests need.
Some other threads you could reference, pay special attention to this if you want to run the tests in unattended way.
Jenkins on Windows and GUI Tests without RDC