I have a CodedUI test suite. I'm attempting to use a B-D-T workflow in TFS 2015 R2 to deploy the test .dlls to a machine group and execute them.
The tests rely on a couple files - an html start page and an XML file.
First, I attempted to have the tests just run from the bin folder OR where they're deployed onto the machines in the machine group. No suggestions I found on SO or on the first 3 pages of google worked. Whenever I ran a test locally it would copy the test dlls to the TestResults folder and execute from there.
Then, I attempted to use a .testsettings file to deploy the files with the .dlls into the TestResults folder. I still truly don't understand why this is the best or only solution - if I can just execute from the Bin please let me know how, but it does work - locally.
Now in TFS 2015 when I deploy my testing software to the test agent(Member of the machine group), I can see my supporting files exactly where they should be in the test drop location. Then, the test starts. A temp folder is created in a byzantine region of AppData, where only the .dlls and config are copied - not the files specified in the .testsettings file! The testsettings file is specified in the "Run Functional Tests" task, which I reached by browsing to the correct file in source control.
I can provide any information you require. This should be so simple but it has been an enormous headache.
In the configuration you need to setup where you copied the files to on the remote machine using a "Windows Machine File Copy" task executed by the agent. You need to make sure all the files needed to run are copied to the server.
Then in your .testsettings file specify which of the files you copied over you want to deploy to the temporary test context directory. Only files already copied to the test agent can be selected here.
The File copy task would look something like:
Related
I seem to be getting a lot of pain with the processing of app.config and token files (we have this working with the older ".11" templates).
It looks like currently (using the ReleaseTfvcTemplate.12.xaml) this is running the tokenisation after the build.
While I can make the app.config / myapp.exe.config stuff work by deliberately copying the .token file into my output folder (so the recursive search finds it) this feels pretty horrid.
As a fix I'm tempted to move lines 182-230 to just before the RunMSBuild task on line 175 (creating a new sequence at that point)
Is this the correct approach or have I missed some documentation somewhere (or a later version of the template?)
Thanks guys... Anyway for future reference I did make the change.
However I had misunderstood the exact nature of the order things happen out of the box which is as follows:
Get the project out of source control
build with msbuild
Copy the .config.token file over the .config file. This is in the TFS template
As part of the deploy to a server then the token entries in the .config files are replaced. This is in the release manager template.
Tests are run in the msbuild binary output folders.
The problem is this doesn't really work if you're using an project type that uses app.config file as the msbuild process renames these output.exe.config during the msbuild stage so you need to create both an output.exe.config (marked as copy to output) and an output.exe.config.token so when the post deploy is the final output gets configured correctly. This also a problem if you want to tokenise some mstest dlls as these typically use an app.config as well. Basically this is a bit of a mess unless you are using web.config.
We worked our way around this by using the modification I suggested above (you need to create a sequence at line 175 and move lines 178-230 up into the sequence, this is GetBuildDirectory variable bits and the if statement) followed by adding an additional deployment stage which copies back onto the build server with the new tokenised files so the mstest can run against them.
So our new process looks like this:
Get the project out of source control
Copy the .config.token file over the .config file i.e app.config.token copied over app.config
build with msbuild (this means we end up with tokenised myapp.exe.config and mytests.dll.config)
As part of the deploy to a server then the token entries in the .config files are replaced. This is a release management step in the release template.
Deploy the tests back into a folder on build server (think this has to be a fixed folder until update 4 of release manager is deployed) the token entries in the .config files are replaced (so our integration tests can use the newly deployed servers). This is a release management step in the release template.
Tests are run in the fixed folder on the build server (rather than the msbuild output directory) so the test wildcard needs to be changed in the tfs build template.
Quick final note we don't use that build directory variable and it's left at blank I'm not convinced this would work if it was set to a value...
The replacement of variables in config files with Release Management happens at deployment time and not at compile time.
When RM deployes your app it inserts the correct variables.
It sounds like you're hitting one of two issues:
You need to include the .token file in your project and make sure it's set to Copy Always, so that it gets copied to the build output folder.
If you're building a web application, I've seen a bug in the release build process template that doesn't touch the contents of the _PublishedWebsites folder. I don't know if it's been fixed in Update 4 or not, but it was definitely still a problem in earlier versions.
I would like to create a TFS 2013 build definition, which builds a SQL project and outputs the complete CREATE DATABASE script to the bin folder.
When I compile my project on the local machine, the SQL script is output to the bin[configuration] folder correctly as per the database project settings.
However when I build the same project using TFS build automation, the build succeeds but no bin folder is created and no sql script is output.
Any ideas what I'm doing wrong?
Additionally, when the script is output locally, the paths are blank:
:setvar DefaultDataPath ""
:setvar DefaultLogPath ""
Any idea how I can populate these for SQL Server 2008 r2 defaults?
Edit:
Actually, if I set the build to target only the .sqlproj file, then the bin is output correctly. The problem comes when the sqlproj is built as part of a solution with other projects.
The build agent does not create individual bin folders in the src working directory; only the start-up project's output is produced and copied to the final bin folder.
I would like the build to generate the sql script and copy it to the output folder along with the other project outputs.
Is this possible?
I ended up going with a workaround based upon This Answer.
If I build the database project as part of the solution, the script does end up in the drop folder, just not the build agent's working folder.
Using:
<CommentOutSetVarDeclarations>True</CommentOutSetVarDeclarations>
in the publish profile allows me to set the default path etc. from the command prompt which was an acceptable alternative in my case.
I am using Chutzpah to run our Jasmine tests.
I have added the Chutzpah dlls to the solution and updated the build to run *.js tests.
The project structure is as follows:
MyApp.Web
Scripts
App
Home
DateControl.js
MyApp.Web.Tests
Scripts
Jasmine
lib
Chutzpah (dlls)
Spec
App
Home
DateControlSpecs.js
The Jasmine test file uses a reference tag to reference the file to be tested
/// <reference path="../../../../../../App.web/scripts/app/home/datecontrol.js" />
The Jasmine tests are run however I get the following error:
ReferenceError: Can't find variable: dateControl in file
dateControl is the object under test.
If I copy the code to be tested into the Jasmine test file then the tests pass.
The jasmine tests with the reference tag pass using the Chutzpah Test Adapter in Test Explorer in Visual Studio 2013.
I have no idea why this is not working.
Update
#jessehouwing's answer pointed out the route of the issue for me.
The build folder on the build server has the following structure
bin
_PublishedWebsites
src
The jasmine test scripts and libraries are copied to bin whereas the script files under test are copied to both the src directory and the _publishedwebsite directory.
I am looking for an easy to manage solution so that any future devs do not need to know they need to link any new scripts into the test project.
Solution I can live with for now
For both the jasmine test files and the files under test I added "Always copy" in properties as per the Chutzpah documentation. I didn't realise that the files under test were actually being copied to the builds bin folder along with the jasmine test files.
So on the build server I ended up with the following struacture
bin
scripts
app
* application js files
specs
* jasmine test files
In my jasmine test file I add two references:
The first to the Visual Studio directory sturcture of the file to be tested
The second to the build bin folder structure
Hack 1
I can get it to work by adding another reference to my jasmine test file which uses the structure of the build directory and not the solution directory structure but again this is open to error as the path may be typed in incorrectly.
The second reference below will allow the tests to run in both VS Test Explorer and TFS Build
/// <reference path="../../../../../../App.Web/scripts/app/home/datecontrol.js" />
/// <reference path="../../../../../_PublishedWebsites\App.Web/scripts/app/home/datecontrol.js" />
Hack 2
By adding an xcopy command to the post-build build event of the project I can copy the files to be tested into the same location as the jasmine test files.
This is most likely caused by the fact that Team Build separates the Sources and Binaries folders and enforces a different output structure in the Binaries folder as well. This means your files are not in the place you expect them to be.
To ensure the test runner can always find your references you must use the "Add as Link" option in Visual Studio and set their build action to "Copy Always" so that they're copied to the test directory upon test execution.
Plus, you might need to enable "Deployment" in the Test configuration in your Build Definition, as outlined here.
The default behavior of TFS Build Server is copy all output, dlls, exes to the folder C:\Builds\{Build_Agent_Number}\{Team_Project_Name}\{Build_Controller_Name}\Binaries. I have customized the build process, not to copy to that folder but instead copy it to another, let say C:\Builds\{Build_Agent_Number}\{Team_Project_Name}\{Build_Controller_Name}\Sources\Some_Folder\Some_Sub_Folder.
The build process works as expected but the problem is all tests don't run. I debugged the build process template and found that it could not find test dlls. I want to know how can I specify the test dll location in the testsettings file or in the build process template.
You can set it on Find Test Assemblies activity in you Build Process Template workflow. It locates a little below the middle of the process.
Navigate to Find Test Assemblies activity inside Run MSTest for Test Assemblies.
Pree F4 to bring up the properties windows then change MatchPattern to the root path that contains assemblies you want to test. For example, String.Format("{0}\{1}", MyCustomAssembliesPath, testAssembly.AssemblyFileSpec)
I’m using a few custom MSBuild tasks that are checked into source control. I would like to import these tasks into my TFSBuild.proj file that TFS uses to build the project. Right now I have created a 2nd project file that includes all of the uses of these custom tasks. I do this because I can run this project file after the workspace has been created and the files have been downloaded.
Is there a way to reference the files on the server from the build project so that I don’t have to do this 2 step process?
I wish the following would work.
<Import Project="$/My/Server/Path/Custom.tasks.targets" />
This question is related to another question. In my case the tasks are checked into source control so that the build machines don't have another required install. Placing the tasks locally doesn't work very well in this case.
If you put the tasks and the .targets file(s) in the same version control folder as your TFSBuild.proj file (or in a sub-directory beneath it), TFS Build will download all of that content to your build agent prior to kicking off your build script.