I have some integration tests that hit a webserver and verify certain functionalities. Depending on the build environment, the server will be at a different address (http://localhost:8080/, http://test-vm/, etc).
I would like to run these tests from a TFS build.
I'm wondering whats the appropriate way to configure these tests? Would I just add a setting to the config file? I'm doing that currently. Incidentally we do have a separate branch per test environment, so I could have a different config file checked in for each environment. I wonder if there is a better way though?
I'd like the build project to be able to tell the test what server to test. This seems better because then I don't have to maintain config information on a per branch basis.
I believe I'd be using NUnit for Team Build (http://nunit4teambuild.codeplex.com/) to get NUnit/TFS to play together.
I can think of a couple options:
Edit the .config file via command line before the test runs.
If the setting depends on which machine the test is run from, you could put it in machine.config
Related
I'm just getting started with Jenkins and I have a few doubts that must be silly, but I'm stuck at it.
After I build my project Jenkins save the build file in some specific path?
Using Ranorex for automation test, is it better to put my files locally on the server or push them to a repository?
Note: I just start tried to use this, at this moment I can check for changes at BitBucket, build the file, build the Ranorex test and run the test.
Jenkins is quite a versatile application that allows system setup to specific needs and requirements of the test project. So i'd say go with the way that seems most logical/easiest. It's kind of a learning process as well so you will be able to understand the working flow of Jenkins itself.
But to answer your 2 questions:
1) By build files i believe you mean the test reports? - For this I actually use the Jenkins UserContent folder. This requires the "Copy to slave" plugin to be installed. With this you will get an additional Post-build Action where you can specify the files that will be copied over to the UserContent folder. But don't forget to specify a common layout for the naming of report files through the Ranorex run parameters ("/rf"). The UserContent folder actually acts as a web server and you can directly link the URLs for email reports. (eg. "http://Jenkins-server.com/UserContent/Regression-Client-Test-#1.html")
2) This totally depends on the system setup. But i can give you an example on how our system is currently set up. So we have Jenkins which runs on a Linux machine. It is only used to manage and run the tests and the actual machine does not include the automation test project. Then we have the test machine which runs on Windows and holds the actual automation tests. This machine is connected to Jenkins through the Slave functionality. So basically when someone starts a test job Jenkins from the Linux machine sends a command to the slave to start the automated tests. When the test run has finished post-build actions take over and copy the report files from the Slave machine to the Linux machines UserContent folder.
Now when talking about the test project management. It's a good idea to use a gir repository which will add another layer of somewhat "security". But if you have a small team (or you are the only test developer) then there is no actual need for it. You just copy the project to the test machine to a specified folder whenever needed/updated and you are ready to run it.
Regards,
Martin
I have a VM where MS CRM is installed and can be access using http://localhost:5555/Orgname/main.aspx.
I have created Unit test cases in my VM by refering the Organization Url as
http://localhost:5555/XRMServices/2011/OrganizationService.svc?wsdl
When I build the test project it connects to CRM and executes the test methods without any error.
Where as when I do a check-in ,the build is getting failed due to the reference to url "localhost".
for Build we have a separate Build server.
Can any one let me know how to solve this.
Your tests are being executed on the build server and it looks like some of your tests are of the Integration kind and not of the Unit kind, as such it's looking for a configured CRM instance on that server (localhost resolves to the host itself for every machine), and can't find any. Which means you have a few options:
Install CRM on the build server, extend the build process to deploy CRM to the build server during build in order to run your tests. A build process like the one developed by Wael Hamze can be extremely helpful for such a solution.
Do not include a localhost address, but actually check in a location that points to a shared dev environment, the build server can connect to. This is not ideal, as the build may be dependent on specific data being present and concurrent builds may break due to strange interactions. If you configure the build agent to only run 1 concurrent builds, it may work.
Disable the tests that depend on an installed version of CRM. You could put a [TestCategory("Integration")] on these tests and then set a filter condition on the build to exclude this test category.
Or you could try to improve your tests by making them independent of your configured instance, using Fakes or any other mocking framework. There are several testing frameworks specifically made for CRM workflow activities and other parts specific to CRM.
You need to remove the Integration Test from the list of Unit Tests that are executed as part of the build. I recommend creating a new project called [projectundertest].IntegrationTests and adding all your integration tests there. Then configure the build to only execute UnitTests...
You build server is trying to execute all of the tests in your solution. Unfortunately you have an Integration Test masquerading as a Unit Test. When it is executed the test tried to access CRM and fails. This is correct behavior.
In our TFS 2013 project we have a set of simple MSBuild based integration tests alongside our unit tests which test stored procedures and other logic which need a database server to be present, for example
[TestMethod]
[TestCategory("Integration")]
public void SomeTest()
{
InitialiseData();
var results = RunStoredProcedure();
AssertResultIsCorrect(result);
}
As you can see we have tagged these tests as "Integration" so that we can supply a test case filter that excludes these tests when we don't want to run them. These tests are all designed to run against an installed copy of our database (we have a process which deploys an up-to-date copy of our database to a SQL Server instance).
What we would like to do is create an integration test build definition in TFS which we can schedule to deploy our database and then run all of these tests against a specific SQL Server instance.
At the moment we use the standard TFS build process with two test sources - the first runs a "unit test" which installs the database, the second contains all of the actual integration tests. The problem we have is passing the connection string & database name into the unit tests from the build definition. Currently the connection string is in app.config files for each of the test projects, however this is less than ideal as it means that we are constantly getting failing test runs, either due to developers checking in the wrong connection string, or running tests locally against the build database at the same time that a build is running. This setup also limits us to running one build at a time.
Is there a way that we can specify the connection string and database name to use as part of the build workflow template instead?
With a combination of SlowCheetah for your config transformation and VS linked files, I think you can solve this (and based on the OP you probably already have :). Make a new solution configuration in your solution for the scenario you described. This solution will not be used on dev machines, only by TFS build definition (under Process, Items to build, Configurations to build).
The Configuration Manager for the solution would then use the solution configuration only for the test proj.
Add your SlowCheetah transform for your new solution configuration and put in your db conn string you need for TFS for that new transform.
Now in the tests project, copy over all the config files as linked files. This will allow the test executions to respect the test config file that SlowCheetah will transform. You may have to adjust your configuration reading in your test proj(s) to account for this.
This isolates the solution configuration to only the TFS server since only it will be building with your new solution configuration. Now TFS will have a config file that points to your specific TFS database connection that no other machines respect.
The background is : I am trying to implement an automated integration test solution. I want to have a FitNesse server running which QA/Users can maintain the test cases. During our nightly build, we want to have the test run locally in the build machine. (In our build script, we are going to startup Jetty, and FitNesse test cases are invoking the RESTful APIs)
When I am looking into the fitnesse-maven-plugin (http://mojo.codehaus.org/fitnesse-maven-plugin/), in the description of goal fitnesse:run, it said that:
This goal uses the fitnesse.runner.TestRunner class for calling a remote FitNesse web page and executes the tests or suites locally into a forked JVM
However, when I am using this plugin with FitNesse version 2009xxxx or 2008xxxx (with a special patch of this maven plugin), I found that the test is not running locally. Instead, I saw new test results created in the remote FitNesse wiki server.
May I know if it is due to change of behavior of FitNesse? (Coz the fitnesse maven plugin is depending on a much older version of FitNesse) Also, with the original Test Runner being deprecated, is it possible to have the behavior I am looking for? (Pages defined in remote server, but run locally in build machine)
Or, is such way of work no-longer a recommended approach to use FitNesse? (If so, I will need to change the approach of the automated test)
One solution I've used is the wiki import option feature. This can import the latest changes from the remote wiki to your local build server.
http://fitnesse.org/FitNesse.UserGuide.WikiImport
You can also tell it to auto-update when running the tests rather than having to re-import manually whenever they change.
Another possibility is to use a source control plugin to automatically commit changes by QA/Users from the remote wiki and pull them down as part of your build.
I have a team foundation server with build server, when I run a build it deploys to a website on that box. However I also want to do the same on Production which is a server on an external network and not part of the same domain.
I thought about looking at TFS Deployer but that just seemed to work within a network, I'm going to test it out as soon as I get a chance but I thought the best idea was to ask here when working with something so critical.
Is it a really bad idea to have a way of easily deploying to production?
Does anyone here deploy to production using whatever method? How do you do it?
Essentially the accepted answer will go to the person who can tell me the best method for achieving a deployment but pointing me in the right direction is sure to get an up vote as long as it's not too obvious.
Depending on the infrastructure you have available to you you can use wix to create msi's and use SMS configuration manager to deploy them to a target collection. This is the direction that we are moving to but have not reached yet. We also integrated wix into our build process to create the MSI artifacts. The reason we wanted to go down this path was because we are using CruiseControl.net as our continuous integration server and we have a nant script that we use to perform both the build process and the deployment process. They are both separate targets in the nant file, but what we wanted was a consistent model of deployment to all environments including production.
What are are doing currently is we are manually moving zips (which are artifacts of our current build process) to production. When the zips are unpackaged in the production environment we have to remove all the web.config, app.config etc from the zips and if we have new entries in the configs they are made manually.
Found msdeploy http://blogs.iis.net/msdeploy/archive/2008/01/22/welcome-to-the-web-deployment-team-blog.aspx