I am new to the world of scripting with TFS2015. I created a script that builds all of the projects within my solution (it is a rather large solution) and puts it out in a shared folder (where each project has its own subfolder).
I would like to create a separate script for each project that simply copies the bin folder from the shared and pastes it out on my Test environment. I rarely need to deploy everything, so the idea is one build...multiple deploys.
However, when I run my deploy script using the Copy Files step it is doing another build. Although it copies the files that I expect, it is after a full build that creates the folder structure for the build.
Am I able to make the Copy Files step NOT do a Build?
Here is the steps that my script is curently doing:
As you can see, there is only one step (Copy Files) but it still does the Get sources and copies everything into a new folder on the build box like so (where the number keeps incrementing up with each run of the script):
I just want to copy the files from the Source to the Target and not do a build or Get Sources.
It looks like you're still on TFS 2015 RTM or Update 1. Which is already pretty old technology if you compare it to the lifetime of the new build system which was introduced with this version.
TFS 2015 update 2 has introduced a similar system to the Build pipelines to orchestrate Releases. This doesn't require you to map any workspaces or git repositories and can act on the artefacts of your builds or simply on the contents of file shares.
It makes sense that a Build has to build something and in order to build something, it has to get the things to build. If you're actually not building something, then you're probably deploying or releasing or packaging something else. Hence the distinction between Build and Release pipelines.
TFS 2017+ has an option to disable the syncing of sources. Primarily to allow people to get the sources themselves in creative ways (e.g. a custom powershell script that invokes git.exe).
My primary advice would be to upgrade to TFS 2018 update 3 or at least TFS 2017 update 3.1, worst case TFS 2015 update 4.1. The fact that versions older than update 2015.4.1 have a known XSS scripting security bug may be reason enough to convince your organisation to perform this update.
Barring that option you're left with one solution:
Link your build definition either to a git repository with only a single commit (If I remember correctly the 2015 agent still crashes when syncing an empty Git repo) or link it to a TFVC repository and set the workspace settings to cloak everything. This essentially causes the build to sync an empty folder, which it can cache, before calling your powershell script.
Related
When we do a TFS Build, can we alter the build output so that the output is limited to only the changes so that deployment payload is reduced ? Example:
When I build a solution, I should only get the changed dlls not all (which includes Microsoft and other 3rd party dlls which are never changed.
Configure CI solution in TFS 2015, and unchecked clean options, Since TFS 2015/2017 always delivered all files - changed and unchanged, but I need only changed. This trick doesn't solve the issue:
Build (TFS Build), only what is changed
Followed a couple of other sources.
IncrementalBuild property in TFSBuild project
Incremental builds in TFS
Applied these tricks to update project with few settings (IncrementalBuild =True, ForceGet=False, SkipInitilizeWorksplace=True, SkipClean=True) under PropertyGroup definition to the end of the TFSBuild.proj file.
But the issue still persists, we are unable to produce only changed binaries in build folder, there is always all files.
Please help me to achieve the desired build output.
Incremental builds only rebuild assemblies that don't depend on changed files. But it does copy all of the project output (subsequent projects that depend on it may depend on these assemblies and files being there).
This causes incremental builds to be much faster, but it doesn't "only deliver the changed files". It always delivers all files whether they are changed or unchanged. On top of this, you could have multiple agents and each agent can have multiple working folders, the incremental build could use any of these as base for the incremental builds, there is no guarantee that the changed files are between your previous build and the current one.
You'll have to implement this feature yourself, it has never been part of MsBuild or TFS Build. It would involve querying TFS for the last drop folder and performing a compare after running an incremental build. Then copying just the changed files and a log of deleted files.
PS: The TFSBuild.proj type builds are very deprecated. They have been surpassed by the XAML builds in TFS 2010 and have been considered "legacy" since then. They have subsequently been surpassed by the new VSTS/Azure DevOps build system which has deprecated the XAML builds. Most of the properties that interact with Source Control are ignored when a TFSBuild.proj project is executed in the Legacy XAML workflow. Instead, the XAML agent takes care of fetching the sources prior to passing control to MsBuild. These new VSTS/Azure Devops build tasks are now also getting YAML support for Git based source control repositories.
When we do a TFS Build, can we alter the build output so that the output is limited to only the changes so that deployment payload is reduced ? Example:
When I build a solution, I should only get the changed dlls not all (which includes Microsoft and other 3rd party dlls which are never changed.
Configure CI solution in TFS 2015, and unchecked clean options, Since TFS 2015/2017 always delivered all files - changed and unchanged, but I need only changed. This trick doesn't solve the issue:
Build (TFS Build), only what is changed
Followed a couple of other sources.
IncrementalBuild property in TFSBuild project
Incremental builds in TFS
Applied these tricks to update project with few settings (IncrementalBuild =True, ForceGet=False, SkipInitilizeWorksplace=True, SkipClean=True) under PropertyGroup definition to the end of the TFSBuild.proj file.
But the issue still persists, we are unable to produce only changed binaries in build folder, there is always all files.
Please help me to achieve the desired build output.
Incremental builds only rebuild assemblies that don't depend on changed files. But it does copy all of the project output (subsequent projects that depend on it may depend on these assemblies and files being there).
This causes incremental builds to be much faster, but it doesn't "only deliver the changed files". It always delivers all files whether they are changed or unchanged. On top of this, you could have multiple agents and each agent can have multiple working folders, the incremental build could use any of these as base for the incremental builds, there is no guarantee that the changed files are between your previous build and the current one.
You'll have to implement this feature yourself, it has never been part of MsBuild or TFS Build. It would involve querying TFS for the last drop folder and performing a compare after running an incremental build. Then copying just the changed files and a log of deleted files.
PS: The TFSBuild.proj type builds are very deprecated. They have been surpassed by the XAML builds in TFS 2010 and have been considered "legacy" since then. They have subsequently been surpassed by the new VSTS/Azure DevOps build system which has deprecated the XAML builds. Most of the properties that interact with Source Control are ignored when a TFSBuild.proj project is executed in the Legacy XAML workflow. Instead, the XAML agent takes care of fetching the sources prior to passing control to MsBuild. These new VSTS/Azure Devops build tasks are now also getting YAML support for Git based source control repositories.
Related to TFS 2017 release management artifact files from version control
I'm asking a new question because I believe I have an edge case the answers don't directly address and I don't want to derail that OP. Specifically, how do I allow an independent, offsite team building required supporting scripts in a separate TFS Team Project supply their scripts as an artifact in the Release definition of a separate TFS Team Project? The separate team projects are built by independent customers and we are not allowed to append content to their source control. Further, updates to the scripts must automatically spread to all Release definitions using them on the TFS.
We have about 40 team projects in TFS all running on different schedules. A separate operations team handles all build and release management tasks in TFS.
Because of the constant bouncing between team projects and because ops also wanted to use the version control and work item tracking features in TFS, we created a separate team project for them to store scripts, installers, and license files. These are referenced in other projects' RM tasks for automatic installation/execution. There is also a separate version control folder tree for tracking project specific scripts - like this:
Common
Applications
App1
App2
...
App43
This makes it significantly easier for them to manage their scripts and associate them with work items themselves without having to shuffle across all the other team projects. The dev teams do not have access to the ops project.
However, when linking a version control artifact in RM from their project, it will only bind to the root and appears to copy the entirety of the version control structure to the agent, even though most of this content is not relevant to the app being deployed.
Is there a way to add specific, not all, folders from their project in version control as artifacts to a release definition in a separate project? We have our QA release start the process to production and it pulls in the artifacts from the ops project and the project being released. All subsequent releases reuse the artifacts that succeeded in the QA build instead of going back to the server for new versions of the artifacts.
Build definitions don't let us pick workspace paths outside of the team project so I don't see a way to pull in their scripts in a build step, either.
Is there a way to do this? How are other organizations handling this issue?
No.
The same answer I provided in the other answer applies here: Don't. Publish them as NuGet packages or as separate build artifacts; a release definition can have multiple artifacts linked to it.
I appreciate Daniel's answer and I believe what he is stating is best practice. However, I believe I found a more direct technical answer to my question through the use of additional repositories.
Release Management allows you to reference Git repos and branches independently, like I had originally hoped to do with folders under the TFVC repo already in a separate project. In this way, we configured the TFVC repo to handle large binaries (installers), license files, etc. which we version and put in a Team Project Nuget feed for reference from RM. To address the folder issue, we created separate Git repos for our operations team project in the same TFS project. Like this:
Binaries (TFVC-based repo)
Git Repositories
CommonDeploymentScripts
Environment Scripts
App1 Scripts
App2 Scripts
etc.
This way TFS RM from any other project can be configured to pull in any one or multiple of these repos as artifacts for use by the agents, bringing down only those scripts that were placed in them.
Also, the ops team doesn't have to cross reference app-specific scripts while bouncing around in a bunch of independent team projects. Note: Daniel is right when he says app-specific stuff should really be versioned and stored with the app project itself. However, some environments may not yet have that luxury so this can fill that need.
RM lets you reference branches under a single Git repo as well so this might be overkill. However, we didn't like the idea of branches under a repo not really having any business ever being merged up into the master - felt like too much room for error.
I'm currently setting up continuous integration using TFS/Visual Studio Team Services (was VS Online), and I'm using the Team Foundation Build 2015 tasks. So not the XAML builds.
I'm using it to build a Xamarin Android project, but that's pretty irreverent I guess,
The process should be like this:
After a check-in:
TFS should download the sources
TFS should increment the version number within AndroidManifest.xml
I've managed to do this by making a PowerShell script for this.
After the AndroidManifest.xml file is modified, it should be committed back into the TFS repository
Then the rest, build deploy into hockeyapp etc
The first steps are all configured, but I'm struggling with the commit part. How do I get TFS to commit the file? I don't really see any task suitable for it. I've tried using the Copy and Publish Build Artifacts Utility - But that did not seem to work, and I'm not even sure if that's the right utility
I'm using the default hosted build agent btw. Any help would be appreciated
Warning
I do want to point out that checking in changes as part of the build can lead to some features of VSTS/TFS not working. Association of work items to the checkin, sources and symbol generation, tractability from changes to build to release and integration with Test Manager, remote debugging, will likely not yield the expected results because the Changeset/commit recorded in te build may not match the actual sources. This may lead to unexpected funny behavior.
Also, if any new changes have already been committed/checked-in after the build has started, the version number may be updated in Source Control for code that was not actually released under that version.
So: First of all, it's considered a bad practice to change the sources from the build process.
Alternatives
There are better ways of doing it, one is to use the build version (Build_BuildNumber or Build_BuildID variables). Alternatively you an use a task like GitVersion to generate the semantic version based on the branch and tag in your git repository. That way your build will generate the correct version number and will increment the revision in case the same sources are built multiple times.
I understand, but I still want to check in my code as part of the build
If these things don't work for you and you still want to check in the changes as part of the build, you can either use the TFVC Build Tasks if you're using TFVC or use the Git Build Tools to add the remote to the local repository and then use the git commandline tools to commit and push the changes back to the repository.
These extensions require TFS Update 2 to install. But you can push the individual build tasks using the tfx commandlien tool. For the TFVC tasks the process is explained here.
On mac
On the mac it's going to be harder since you're using TFVC. My TFVC tasks leverage the TFS Client Object Model and Powershell to communicate to the TFS Server. The tf.exe tool doesn't even work on windows when you're in the context of a build, which is why I need to call into the VersionControlServer object directly. Given I'm dependent on these technologies, the tasks won't run on a Mac or Linux agent.
You could try to see whether the Team explorer Everywhere X-platform commandline works from the build agent (using a shell script). I have no way to test this on an actual Mac.
Given the cross platform nature of your project I'd recommend to move to Git, it integrates into XCode and Android Studio, making it easier to do a native UI or build on top of native libraries.
Alternative 2
You could setup a build which does the required changes to the code and then checks in the modified code. Then have a (CI) build run the Android and the Mac builds using the modified code.
I'm using TFS 2013 on premises. I have four build agents configured on a Build machine. Several build definitions compile ASP .NET websites. I configured the msbuild parameters to deploy the IIS application to the integration server, which sits out there in Rackspace.
By default webdeploy does differential deployments by comparing file dates. In my case that's a big plus because copying files from our network to Rackspace takes quite some time. Now, in order to preserve file dates the build agent has to compile the same base set of source code. On every build only the differential source code yields a new DLL, minimizing the number of files deployed.
All of that works fine, with a caveat: a given build definition has to be assigned to a build agent (by agent name or tag). The problem is I create a lot of contingency when all builds assigned to the same agent are queued up. They wait in line until the previous build is done.
In an ideal world any agent should be able to take care of any build, but the source code being compiled has to be the same, regardless of the agent.
I tried changing the working folder of all agents to point to the same location but I get an error because two agents can't be mapped to the same folder. I guess there is one workspace per agent.
Any ideas?
Finally I found a way to do this. Here are all the changes you need to do:
By default the working folder of each agent is $(SystemDrive)\Builds\$(BuildAgentId)\$(BuildDefinitionPath). That means there's one working folder per BuildAgentId. I changed it so that all Agents share the same folder: $(SystemDrive)\Builds\WorkingFolder\$(BuildDefinitionPath)
By default at runtime the workflow creates a workspace that looks like "[BuildDefinitionId][AgentId][MachineName]". Because all agents share the same working folder there's an error trying to create each separate workspace. The solution to this is in the build definition: Edit the xaml and look for an activity called "Get sources from Team Foundation Version Control". There's a property called WrokspaceName. Since I want to have one workspace per build definition I set that property to the BuildDetail.BuildDefinition.Name.
Save your customized build template and create a build that uses it.
Make sure the option "1. TF VersionControl/1. Clean workspace" is set to False. Otherwise the build will wipe out all the source code on every build.
Make sure the option "2. Build/3. Clean build" is set to false. Otherwise the build will wipeout the output binaries on every build.
With this setup you can queue up the same build on any agent, and all of them will point to the same source code and bin output. When the source code changes only the affected binaries are recompiled. I have a custom step in the template that deploys the output files to IIS, to all the servers in our webfarm, using msdeploy.exe. Now my builds+deployments take one or two minutes, because only the dlls or content that changed during the build are synchronized to the servers.
You can't run two build agents in the same folder. The point of build agents is to run multiple builds in parallel, usually on separate PCs. If you try to run them on the same source code, then (a) it's pointless as two build of exactly the same source should produce identical results, and (b) they are almost certainly going to trip over each other and cause the builds to fail or produce unexpected results.
If you want to be able to build and then deploy a series of versions of your codebase, then there are two options:
if you queue up multiple builds, then the last one will "win", so the intermediate builds are of no real value. So if you check in New code before your first build completes, you may as well stop the active build and start a new one. you should be asking yourself why the build is so slow, or why you are checking in changes so often that this is necessary.
if each build produces an incremental update to the deployed result, then you need to pass the output of your builds to some deployment agent that is able to diff it against the deployed version and send only the changes to be deployed. This could be set up to gather results from multiple build agents if that would be beneficial.
but I wonder if perhaps your build Is slow because you are doing a complete build each time (which cleans the build folder, gets all the sources, and does a full rebuild), when what you want is an incremental build (which gets the latest changes, compiles only what is affected, and complete quickly). perhaps you should investigate making your build incremental.