I am using Jenkins for my project CI. And several automation test cases developed by Coded UI(C# language) will be run when the new build is deployed. I created a job on the master(windows) that MSTest those test cases(note: with MSTest under cmd, test cases can be run successfully) but saw the below issue:
Error calling Initialization method for test class QuickUI.SmokeTests: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=159018)
Therefore,
1) Need I use Jenkins slave to run test cases?
2) If so, how to config since the dlls needs to be copied to slave firstly?
3) Is there any account or jenkins service configuration needs to be taken care?
Thanks in advance.
Sounds like you need to get mstest setup on your build machine. The easiest way is to use test agents (http://www.microsoft.com/en-us/download/details.aspx?id=38186) when installing chose test agents.
(below is copied from my answer in Coded UI build server integration process)
You going to want to put your CodedUI tests inside an orderedTest. Right Click on the solution -> add -> orderedTest.
You going to want to install https://wiki.jenkins-ci.org/display/JENKINS/MSTestRunner+Plugin it's not a requirement, but it makes working with mstest in jenkins a little easier.
At this point you just need to configure the plugin in jenkins to run your orderedTest. It will need you to point to mstest and the location of your orderedtest.
1) Recommnd to run test on slave
2) To run GUI test, don't run the slave as windows service
3) Theoratically you only need the slave agent running. However the windows account logged in should have access to all the resources that your tests need.
Some other threads you could reference, pay special attention to this if you want to run the tests in unattended way.
Jenkins on Windows and GUI Tests without RDC
Related
Have seen the following graph representing Jenkins pipeline:
git push --> Git repo --> Jenkins CI server --> Maven build server --> Test server --> Deliver build artifacts --> Deploy
Given that is correct, I am struggling to understand how the above different(?) servers work under the hood and I need some clarification so that Jenkins procedures are not just a black box.
Jenkins CI server, Maven build server and Test server are in reality one physical server?
If the answer to the previous question is yes, these 3 servers are different logical servers?
In my understanding and my case (Java Spring project), Maven build server executes mvn install and since pom.xml contains npm install plus npm run test commands, it is Maven build server that executes the UI tests and not the Test server. Am I right?
Does the Test server execute only the back-end Java acceptance tests?
It could be one physical or virtual server server.
no
To use maven you just need to install maven plugin in jenkins. Configure which version of maven you want to use in "Manage jenkins" -> "Global tools configuration" -> add needed maven version.
Depends on which kind of tests you want to run - you can repeat steps from step 3 for the tool you need or you can install needed tools on the jenkins server manually and add it to the PATH.
At the end just use needed tools in your pipeline or other king of job in jenkins.
Without further context, we must put some points clear:
Jenkins may work on multiples nodes (could be real/virtual/pods)
Jenkins is an orchestrator... This means that it uses different tools in a order given by the pipeline. This order is pretty much: git > build > test > delivery > deploy
Testing server are used for install the software an run a variety of tests, pretty much for several teams.
Build servers are used for running commands to build software
With those clear, these are the questions:
1.- Depends on your infrastructure. Jenkins works with executors that may run on master or in nodes. If your Jenkins is just one server, then yes... It is one physical server. If you use nodes, then it is more probable than you may found one node handling the building, and another one the tests. The definite answer lies within your Jenkins and your pipeline
2.- No... Even with a Jenkins master-agent infrastructure, the building and testing occurs within Jenkins.
3.- Depends on your concept of test server. If you define it as a server where you use as a target for testing purposes, then the answer is no... If you define a test server as a machine that executes test, then the answer is yes.
4.- Depends on what you type or test do you have automatized. You can run unit tests, regression tests, smoke test, etc. For example, you may have some unit test for your back-end, and have some test in karma for your UI. Again, in your case you must check on your pipeline, and the code to check what kind of tests are you running.
Selenium test (testNg) with mvn test from jenkins ,
I cant view the action performed on the browser, it is possible to view the real browser opened and performing action?
I suggest using SeleniumGrid.
You can then let the test run on a node, where you can see all the browser actions. For demonstration purposes, distributed and independent testing I did similar with teamcity for a regression-test project.
Basically there are two steps to setup the environment:
Start the grid for example as background-service.
Start a node which is running within a logged on user context (also as Service or autorun)
optional: Start another node for running tests in background
In your tests the creation of the the WebDriver instance changes like this:
WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capability);
Here a link for more details to setup the Grid2: https://github.com/SeleniumHQ/selenium/wiki/Grid2
No. In typical setup, tests from Jenkins run on randomly available node. You don't know which node is available at a given point. Also typically Jenkins nodes tend to run headless browsers (like phantomjs) since they don't have displays.
Jenkins tests are meant to be run without any manual intervention, like on nightly basis for example. You develop tests locally, see how they run in a real browser and then push it to your main suite so that Jenkins can run it for you. If you want to run tests from Jenkins and view test running live then look for paid solutions like saucelabs or browserstack.
You can using BrowserStack. We have recently released a Plugin that allows you to view your test results right inside Jenkins. It's a paid tool of course :(
In Windows you should not install jenkins as windows application (because in this recent version, Microsoft decided to give services their own hidden desktop even you enable the functionality "interact with desktop" in jenkins service), you may have to deploy it from a war file as follows:
1) Download jenkins.war from Jenkins official site
2) Deploy it by the command prompt : java -jar {directoryOfJenkinsFile}/jenkins.war
3) Now you can access jenkins administration on http:// localhost:8080
Hope that helps you !
I would say that my problem rather lack of information and I need some confirmation than a real problem. It seems somebody else had similar question question.
I put together a machine (Windows Server 2012R2) for POC reasons where a Jenkins installed and it executes Selenium UI tests using nunit. The nunit tests are generated by Specflow.
I could do:
install jenkins
jenkins run by a valid user not by Service account
set up jenkins properly
it can pull the source code from TFS-GIT
it can compile the C# solution
it can execute the test project
the test results are correct
Selenium plugin installed on Jenkins but I don't think it is used in this case because the text execution is about executing nunit and it deals with everything else.
At the moment I don't need the capability to delegate test execution to other Jenkins slaves or machines because the Jenkins does have only one compile task. Compiling, executing and test running can go parallel, the machine able to deal with it.
But, when I log in the server where the Jenkins runs and I watch what happens during CI build (compile and test execution) I can't see that the browser (Firefox) starts, however, the test results and the logs show that a browser was executed.
What I did so far:
jenkins runs as service, the account is an existing account
If I remote to the machine with the account which is set up for the service, then I can't see the browser will be executed, however, the log shows that something had happened.
My question is that, what the hack is happening when my tests are executed by Jenkins? If I execute the command which is used by Jenkins from console on the same machine then I can see that Firefox starts, does what is programmed in the tests and the results are in the result.xml. Can I accept the result as valid result? Can I somehow set up Jenkins the way the browser really executed (I can believe it when I see it :) )?
I think this is because you run Jenkins as a service. Services do not show up in desktop. Workaround is to run Jenkins or slave from CMD.
Jenkins windows slave service does not interact with desktop
I am running UFT Test Suite manually on the host machine, however i now have the requirement to run the smoke tests as soon as the new build is delivered in jenkins. For that i need to run the test suite automatically using Jenkings. Can someone help me in the direction, how this can be achieved? I have never used Jenkins before.
Looks like theres a plugin for Jenkins which enables integration with HP UFT. Take a look at the Jenkins wiki page for instructions about how to install and use it.
As you didn't mention that you are using ALM so I am assuming that you have some kind of driver script in vbs which runs your test suite. You do not necessarily require any plugin in jenkins to run your UFT suite(however you can also check the suggestion given in the previous answer). Create a job in jenkins and call your vbs file (driver file) or a bat file which kicks off your execution your test machine.
Your test execution machine can be configured as jenkins slave. Only thing which you need to take care off is to remove the settings which can let go your jenkins slave machine which is your uft execution machine in standby mode or locked user mode(uft scripts will fail otherwise). You can handle this by configuring windows settings(if admin allows) or by using some small scripts to move mouse after some intervals.
Your test execution job in jenkins(master) machine can be made dependent on the job which goes to success when new build is available. As soon as new build is available - that job goes to success then your Test job gets notified and connects to UFT execution machine(slave) and starts running your test.
I am trying to integrate ALM with jenkins but not working
Started by user admin
Running as SYSTEM
Building in workspace C:\JENKINSHOME\workspace\CLRP_AUTO_RN
[CLRP_AUTO_RN] $ C:\JENKINSHOME\workspace\CLRP_AUTO_RN\HpToolsLauncher.exe -paramfile props05092020010832244.txt
"Started..."
Timeout is set to: -1
Run mode is set to: RUN_PLANNED_HOST
Unable to retrieve test set folder: Node not found.
Node not found.
Starting test set execution
Test set name: CLRP Smoke Test Automation Run, Test set id: 104
Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG))
Could not create scheduler, please verify ALM client installation on run machine by downloading and in installing the add-in form: http://almserver:8080/qcbin/TDConnectivity_index.html
Build step 'Execute Micro Focus functional tests from Micro Focus ALM' changed build result to FAILURE
Finished: FAILURE
We currently have coded UI regression test suite (UI function test) which runs perfectly using MTM on test anent/controller.
As part of CI our team is using Jenkins. Here we want to integrate coded UI test with Jenkins. Can you please tell me how can I configure this?
Is there a way Jenkins will access the TFS build of my coded UI test project which will have test run setting file (with info of test controller and agent to run test)?
I don't want to run coded UI tests on Jenkins server (but on test agent) as it will need to have a active desktop session.
A great example of running UI with Jenkins is executing selenium tests. This approach suggests you to run browser and simulate user actions. The same you can do with your UI and basically this does not require any additional configuration steps except slave configuration. You need to use Slave Configuration via Java Web Start as only this gives you ability to interact with Desktop:
If you need display interaction (e.g. for GUI tests) on Windows and you have a dedicated (virtual) test machine, this is a suitable option.
Other slave configuration ways may just not run your UI.